DEV Community: Ajinkya Singh

Multi-Core Cache Coherency: The Invisible Lock You Never Wrote

Ajinkya Singh — Sun, 30 Aug 2026 18:53:30 +0000

Sources this series builds on: Chili's (ChiliTomatoNoodle) multithreading
playlist,
CMU 15-213 (Computer Systems: A Programmer's Perspective), various CppCon talks,
and LLMs for restructuring and sanity-checking. Everything here is my own code,
compiled and run — not just transcribed.

Recap — and the question this article finally answers

Parts 3 through 5 built up a single-core picture: bytes and addresses, why
caches exist, and cache lines as fixed 64-byte aligned chunks. This article
introduces the piece that was missing — what happens when more than one
core wants the same memory at the same time — and it's the direct,
mechanical answer to the question this series has been sitting on since
Part 1: why did the mutex-protected version cost 13.2 seconds, over 4x
slower than not threading at all?

The new problem: private caches, shared memory

Registers, L1, and L2 are private per core. Only L3 and RAM are shared
across the whole chip (Part 4). That means the same 64-byte cache line can
end up copied into more than one core's private cache at the same time.

If nothing handled this, one core could write a value and another core
could read the same location moments later and get something already out
of date — not because of a bug in your code, purely because of where the
data happened to be cached. Hardware solves this fully automatically, with
no code from you required, via a coherency protocol.

The four states every cached line is tagged with (MESI)

The most common protocol family is called MESI, named after the four
states a cached line can be in on any given core:

Modified (M) — only this core has it, and it's dirty (differs from memory). Free to read or write. Nobody else has a copy.
Exclusive (E) — only this core has it, and it's clean (matches memory). Free to read; writing it silently flips it to Modified with zero bus traffic, since nobody else needs telling.
Shared (S) — this core and at least one other core have clean copies. Free to read. Cannot write without first evicting everyone else's copy — this is the expensive transition.
Invalid (I) — this core doesn't have a usable copy. The next access is a miss.

Every private cache runs this state machine independently, watching every
other core's traffic to know when to react. You don't configure it or see
it — it's just always running.

What actually crosses the wires

The state diagram shows the outcome. What physically happens on the
interconnect between cores is a small set of message types — every cache
"snoops" (watches) this traffic to see if it needs to react:

Message	Sent when...	Effect on other cores
BusRd (Read)	"I want to read a line I don't have"	holders downgrade to Shared; a Modified holder flushes first
BusRdX (Read-For-Ownership)	"I want to write a line I don't have"	every other copy anywhere is invalidated
BusUpgr (Upgrade)	"I already have it Shared, now I want to write"	every other copy invalidated — no data re-fetch needed, just permission
Flush	"I had it Modified and someone else needs it"	dirty data written back before giving up ownership

Here's the exact sequence two cores fighting over one line generate, every
time ownership swings back and forth:

Every BusRdX round trip costs on the order of the L3/interconnect
latency — tens of cycles — even though the actual arithmetic underneath is
a single cycle. This ping-pong is what's actually being measured whenever
two cores contend for the same line — not the math.

A cost ladder — not every core-to-core interaction is equal

Two cores sharing a line can meet in several different ways, and they are
not all the same price. Cheapest to most expensive:

Scenario	What happens	Relative cost
Local hit — already M/E on this core, no contention	Write happens purely in L1, zero bus traffic	Free
Cold read/write — nobody else has the line	Fetch from L3/DRAM, no one to invalidate	Cheap
Read, another core has it clean	Cache-to-cache copy, no invalidation	Moderate
Write, another core has it Shared (upgrade)	Permission-only request, no re-fetch	Moderate
Write, another core has it Exclusive/clean	Cache-to-cache transfer, invalidate, no flush	Moderate
Read, another core has it dirty	Full flush + cache-to-cache transfer	Expensive
Write, another core has it dirty	Full flush + invalidate + cache-to-cache transfer	Most expensive

The bottom row — write, while the other core is sitting on a dirty
(Modified) copy — is the one that matters most for this series:

Once two threads are both actively writing the same line, every single
write from either side lands in exactly this scenario — because whichever
side wrote last leaves the line Modified, which is precisely the expensive
case for whoever writes next. This back-and-forth, repeating on every write,
is the physical mechanism behind Part 1's 13.2 seconds: 400 million
lock/write cycles, each one paying this full flush-and-invalidate round
trip whenever the line had bounced to the other core since the last write.

The key insight: invalidation happens per line, not per variable

The coherency protocol doesn't know or care that a line contains "your
variable A" and "my unrelated variable B." It only knows this line was
written to — so it invalidates the whole line for every other core
holding it, even if those cores only ever touch a completely different
variable that just happens to live in the same 64 bytes.

Two threads can have zero shared variables in their own program logic —
no mutex, no atomic, nothing that looks like synchronization in the code at
all — and still slow each other down, purely because the hardware treats a
write to any part of a line as a reason to invalidate the entire line. This
isn't hypothetical. It's exactly what's waiting in the next article.

Wait — doesn't this mean coherency already prevents races?

Worth closing a gap explicitly, because it's a natural question once you've
seen how much machinery MESI actually runs: if hardware keeps every core's
cache honest automatically, why did Part 1's cnt += x still race?

Because coherency is a per-transaction guarantee, and cnt += x is not
one transaction — it's three (Part 1: read, modify, write). MESI correctly
guarantees that each individual read or write sees a consistent value.
It says nothing about a sequence of them. Nothing in MESI stops Core B
from sneaking in its own read between Core A's read and Core A's write —
there's no rule saying "once a core reads a line intending to write it
back, nobody else may touch that line until it does." That gap is exactly
where a race condition lives.

Every individual bus transaction above was perfectly coherent — the line
moved Shared → Modified → Invalid → Modified exactly as MESI dictates.
Coherency did its job at every single step. The bug isn't in the protocol;
it's in the gap between load and store, a gap coherency was never designed
to close. This is exactly what the lock prefix fixes and coherency alone
doesn't: it extends exclusive ownership across the whole read-modify-write,
not just one step of it, so no other core's read can slip into that gap.

Glossary added this part

Term	Meaning
Cache coherency	The guarantee that all cores see a consistent, up-to-date view of memory, even with private cached copies
MESI	Modified / Exclusive / Shared / Invalid — the four states a cached line can be in
BusRd / BusRdX / BusUpgr	The actual messages cores send to read, write, or upgrade permission on a line
Snoop	A cache watching bus traffic to see if it needs to react (invalidate, flush)
`lock` prefix	An x86 instruction prefix forcing a read-modify-write to hold exclusive line ownership for its entire duration

Takeaways

Cache coherency is real, automatic, and enforced entirely in hardware via a state machine (commonly MESI) and a small set of bus messages — no code of yours configures or sees it directly.
Not all core-to-core interactions cost the same. Writing while another core holds a dirty copy is the most expensive case there is — a full flush, invalidate, and transfer, every time — and it's exactly the case two threads land in on every write once they're actively contending for one line.
Coherency and atomicity are different guarantees. Coherency keeps a single access honest; atomicity (the lock prefix) keeps a sequence of accesses honest. cnt += x needs both, and plain non-atomic code only ever gets the first one — which is exactly why Part 1's race happened despite coherency running the whole time.
Invalidation happens per cache line, not per variable — two threads with zero shared variables in their own logic can still contend, purely from memory layout.

The 64-Byte Truck: Cache Lines Explained

Ajinkya Singh — Fri, 28 Aug 2026 19:23:55 +0000

Recap

Part 4 established why caching happens at all — spatial locality: if you
access one address, you're likely to want its neighbors soon, so hardware
fetches more than exactly what you asked for. This article answers the
obvious follow-up: how much more, exactly, and why that specific amount?

The answer is a fixed-size chunk called a cache line, and on most modern
CPUs that size is 64 bytes — always, no matter what you actually asked
for. This one fact is what the rest of this hardware track, and eventually
false sharing, is built entirely on top of.

What actually happens when you read one `int`

Say your code does int x = data[3]; and that triggers a cache miss — the
data isn't in cache yet, so it has to come from RAM. The natural mental
picture is the hardware reaching into RAM and pulling out exactly the 4
bytes that make up that one int. That is not what happens.

Instead:

The entire 64-byte block gets pulled in and cached as one unit, and your
int gets read out of that block afterward. You paid for 64 bytes whether
you needed them or not.

Why 64 bytes specifically — two independent reasons

Reason 1: the physical width of the connection

The hardware path between RAM and the CPU (and between cache levels) is
physically built to move a fixed number of bytes per transfer — its "width."
On most modern CPUs that width is 64 bytes. The critical part:
transferring 4 bytes over this path takes the same amount of time as
transferring all 64, because the limiting factor isn't the amount of data,
it's the fixed overhead of doing a transfer at all. Think of it like a
delivery truck that costs the same to dispatch whether it's carrying one box
or a full load — you fill the truck because the trip itself is the expensive
part, not the cargo.

Reason 2: spatial locality says it's worth it

Since nearby addresses are likely to be accessed soon anyway (Part 4), fetching
the surrounding 60 bytes "for free" — same transfer cost as fetching just 4 —
is a good bet almost all the time. Iterate over an array, and this pays off
constantly: fetching element 0 also fetches elements 1 through 15 (for 4-byte
ints), so the next several loop iterations are already cache hits before you
even ask.

Cache lines are aligned, not centered on what you asked for

A subtlety worth being precise about: the 64-byte chunk fetched isn't "the 64
bytes starting at the address you asked for." It's the 64-byte block that
address falls inside, where blocks are laid out in fixed segments starting
from address 0 (0-63, 64-127, 128-191, and so on). That's what aligned
means here.

So asking for the int at address 1004 doesn't fetch bytes 1004-1067 — it
fetches whichever fixed 64-byte segment contains address 1004 (addresses
960-1023, in this example). The requested bytes might sit anywhere inside
that segment — start, middle, or end.

This is the single most important detail in this article. Because lines
are fixed, aligned segments — not "64 bytes around whatever you asked for" —
any two variables that happen to land inside the same aligned 64-byte segment
get fetched and cached together as one unit, entirely regardless of whether
your program treats them as related at all. The hardware has no concept of
"these are two different variables." It only knows about lines.

How much fits in one line

64 bytes holds, for example:

16 ints (4 bytes each)
8 doubles (8 bytes each)
A handful of small struct fields, packed together

So in practice, several "neighboring" variables in a real program — array
elements, adjacent struct fields, even separate global or static variables
that happen to land near each other by the compiler's own layout choices —
very commonly share a single cache line without anyone ever writing code
that groups them together on purpose.

Glossary added this part

Term	Meaning
Cache line / cache block	The fixed-size chunk (typically 64 bytes) transferred and cached as one unit, every time
Bus width	The fixed amount of data a hardware connection moves per transfer, regardless of how much was actually requested
Aligned	Cache lines sit at fixed positions from address 0 (0-63, 64-127, ...), not centered on whatever address you asked for

Takeaways

A cache miss never fetches just what you asked for — it fetches the entire aligned 64-byte segment containing that address, every time.
64 bytes is fixed by hardware bus width (fetching 4 bytes costs the same as fetching 64, so hardware always fetches the full amount) and justified by spatial locality (the extra bytes are usually useful anyway).
Cache lines are aligned to fixed 64-byte boundaries starting from address 0 — not centered on your specific address — which is why unrelated variables can end up sharing a line purely by accident of memory layout.
The hardware has no concept of "your variables." It only tracks lines. Two completely unrelated ints can be neighbors in the same 64-byte block with neither programmer ever knowing.

The Hardware Memory Hierarchy: From L1 Caches to Main RAM

Ajinkya Singh — Sun, 23 Aug 2026 20:08:42 +0000

Recap

Part 3 left one number sitting there deliberately: a register access takes
roughly 1 cycle; a trip to main RAM takes roughly 200 cycles. This
article picks that gap up and asks the obvious next question — if RAM is
really 200x slower than a register, how does any real program run at a
reasonable speed at all? The answer is caches, and understanding why they
work is what makes the rest of this hardware track (and eventually, false
sharing) make sense instead of feeling like memorized trivia.

The problem caches solve

A CPU core executes an instruction in roughly 1 cycle. A single access to
main RAM takes roughly 200 cycles. If every memory access actually went all
the way to RAM, the CPU would spend the overwhelming majority of its time
just waiting — no matter how fast its execution units are.

Caches aren't an optimization bolted on as an afterthought. They're
load-bearing for a modern CPU to be useful at all.

The hierarchy, level by level

Level	Typical latency	Typical size	Scope
Registers	~1 cycle	Tens of bytes	Per core
L1 cache	~4 cycles	32-64KB	Per core (split: instructions / data)
L2 cache	~12 cycles	256KB-1MB	Per core
L3 cache	~40 cycles	A few MB to tens of MB	Shared across all cores on the chip
Main RAM	~200 cycles	GBs	Shared, off-chip

The pattern going down every step: size goes up roughly an order of
magnitude, latency goes up roughly 3-5x, and — this is the detail worth
underlining — what's visible to that level shrinks from "this core only"
(registers, L1, L2) to "every core on the chip" (L3, RAM). That shift, from
private per-core storage to shared storage, is the seed of the
entire cache-coherency problem two articles from now, and it's the reason a
mutex or an atomic operation touching L1-local data behaves completely
differently from one touching something every core has to agree on.

(Numbers vary by CPU generation and vendor — treat the table as
order-of-magnitude intuition, not a spec sheet for any specific chip.)

Why a pyramid, not one big fast memory

Why not just make all of RAM as fast as L1? Two physical reasons, not
engineering laziness:

Speed and size trade off directly in circuit design. The circuitry that makes L1 fast — more transistors per bit, shorter wire distances to the core — is also what makes it expensive and physically large per byte stored. Multi-gigabyte capacity at L1 speed isn't a software problem to solve; it's not economically or physically feasible with current technology.
Physical distance costs real time. Even at near-light-speed signal propagation, a memory bank farther from the core takes measurably longer to respond. L1 is built into the core. RAM sits on separate chips, connected over a bus. That distance alone adds latency, independent of the memory technology used.

So instead of one tier, you get a pyramid: a tiny amount of very fast storage
right next to the execution units, progressively larger and slower tiers
moving outward, with RAM as the big slow tier at the bottom.

How the hierarchy is actually used

When the CPU needs an address, it doesn't go straight to RAM — it checks
levels in order, fastest first:

A successful lookup at any level is a hit. Failing to find it and having
to check the next level down is a miss. Every miss that eventually
resolves — even all the way from RAM — copies the data upward into the
faster levels it passed through, on the expectation you'll want it again
soon.

Why caching works at all: locality

Caching only pays off because real programs exhibit locality —
predictable patterns in what memory they touch.

Temporal locality — if you accessed an address recently, you're likely to access it again soon. (A loop counter, read and written every iteration.)
Spatial locality — if you accessed an address, you're likely to soon access addresses near it. (Iterating an array — element 5 is immediately followed by element 6.)

Temporal locality is why keeping recently-used data around instead of
evicting it immediately is worth doing. Spatial locality is the direct
motivation for the next article: instead of fetching exactly the one
byte/int requested, hardware fetches a whole contiguous block around it —
because spatial locality says you'll probably want the neighbors too. That
block is called a cache line, and it's the single fact the rest of this
hardware track — and eventually false sharing — is built on.

Glossary added this part

Term	Meaning
Cache hit	Requested data found at this level — fast path
Cache miss	Requested data not found here — check the next, slower level
Latency	Time between requesting data and having it available
Temporal locality	Tendency to re-access the same address soon
Spatial locality	Tendency to access nearby addresses soon after one

Takeaways

Without caches, a CPU capable of 1-cycle instructions would spend nearly all its time waiting on 200-cycle RAM accesses. Caches aren't a bonus, they're required for a modern CPU to function at a usable speed.
The hierarchy is a pyramid, not a single fast tier, for two physical reasons: speed/size trade off in circuit design, and physical distance costs real time regardless of technology.
Registers, L1, and L2 are private per core. L3 and RAM are shared across every core on the chip — that shift from private to shared is exactly where multi-core problems start, two articles from now.
Caching only works because of locality — temporal (reuse soon) and spatial (neighbors soon) — and spatial locality specifically is why hardware always fetches more than you asked for.

How CPUs and Memory Actually Work: Bytes, Addresses, and Latency

Ajinkya Singh — Sat, 22 Aug 2026 21:04:53 +0000

Recap — the question this series has been avoiding

Part 1 left a number sitting there unresolved: the mutex-protected version of
complexFunction took 13.2 seconds — over 4x slower than running the
same work single-threaded with no threading at all. Part 2 showed a plain
bool racing across threads and getting hung or skipped entirely
depending on the optimization level. Both articles explained the C++-level
mechanics — read-modify-write, register caching, the compiler's license to
assume no data race exists — but neither one actually answered why a lock
costs that much time or why one core doesn't just immediately see another
core's write.

Those answers aren't in the C++ standard. They're in the hardware underneath
it. This article is where the series stops treating "memory" as a black box
and starts building the vocabulary to actually answer both open questions —
starting from the most basic thing there is: what a variable even is, once
you strip away the syntax.

A computer, stripped down to two things talking to each other

Underneath everything else, a running program is really just two components
in constant conversation:

The CPU — does the thinking. It can only perform very small, very simple operations: add two numbers, compare two numbers, move a number from one place to another. Everything complex your code does is millions of these tiny steps happening extremely fast, one after another.
Memory (RAM) — holds data while the program runs. The CPU itself can only hold a tiny amount of data internally at any given moment (registers, covered below), so anything larger — your variables, arrays, objects — lives out in memory, and the CPU has to go fetch it when it's needed and send it back when it's done.

That fetch-and-send-back relationship is the single most important thing to
carry forward from this article. Almost everything about performance and
multithreading correctness in this series comes back to "how fast can the
CPU get data out of memory, and what happens when more than one core wants
the same piece of memory at the same time."

Cycles — the CPU's metronome

A CPU doesn't work continuously; it advances on a beat. Each tick of that
beat is a clock cycle. On every tick, the CPU can move its work forward
by roughly one small step (a simplification, but the right mental model for
now).

Clock speed — the number on a spec sheet, like "3.5 GHz" — is just how many
of these ticks happen per second. 3.5 GHz means 3.5 billion ticks a second.
So when a later article says "this operation takes 200 cycles," that's not
an abstract unit — it's literally "the CPU sat through 200 of these ticks
before it could continue." At billions of ticks a second that's still a
tiny slice of real time, but it's 200 times longer than something that only
takes 1 tick, and that ratio is what actually matters, not the absolute
time.

Keep this number in your back pocket: reading from a CPU register takes
roughly 1 cycle. Reading from main RAM takes roughly 200 cycles.
That gap — the CPU sitting idle for 200 ticks of its own clock, doing
nothing but waiting for data — is the entire reason caches exist, and caches
are where this hardware track goes next.

Bytes and addresses — a variable is just a name for a location

Memory isn't one undivided blob. It's organized as a long row of small,
individually numbered slots:

A byte is the smallest individually addressable unit of memory — 8 bits. Most things you care about (an int, a char) are a small number of bytes glued together.
An address is the position number of a byte in memory — a house number on a very long street. Address 0 is the first byte, address 1 is the next, and so on.

Here's the reframe worth sitting with: int x = 42; is not, underneath
everything, a special protected thing called x. It's a label your compiler
uses for "start at this address, and the value there is 4 bytes long."
Reading x at runtime means the CPU is told an address and goes and fetches
whatever bytes are stored starting there.

This is the seed of everything coming later in the hardware track: two
completely unrelated variables in your program can end up sitting at
neighboring addresses, purely by how the compiler laid them out — with
consequences neither you nor the compiler intended. That's the exact setup
for false sharing, several articles from now.

Fast vs. slow — the ordering that everything else hangs on

You don't need exact nanosecond numbers yet — just the relative ordering,
because that ordering is the skeleton the rest of the hardware track hangs
real numbers on:

Each step to the right is slower to access but holds more data. That
trade-off — and exactly why the hierarchy is shaped like a pyramid instead of
one giant fast memory — is Part 5.

Cores — why "more than one CPU" changes everything

Modern CPUs usually contain more than one independent execution unit, each
able to run instructions on its own — a core. A 4-core CPU can run four
genuinely simultaneous streams of work, not just fast-switching between one
stream at a time. This is what makes Part 1's speedup real: each thread
lands on its own core, actually running in parallel.

But multiple cores introduce a genuinely new problem that doesn't exist with
a single core: each core typically has its own small private cache. So what
happens when two cores both want to read or write the same piece of memory
at the same time? That question — not anything about instructions or
compilers — is the real, hardware-level reason Part 1's mutex cost 13.2
seconds, and it's exactly what the next few parts of this series are for.

Takeaways

A CPU and RAM are, underneath everything, just two things constantly fetching from and sending data to each other — and that relationship is where every later performance question in this series comes from.
1 cycle for a register, ~200 cycles for RAM isn't a rounding error — it's the CPU sitting idle 200x longer, and it's the entire reason caches exist.
A variable is not a protected, special thing — it's just a name for "some bytes at some address," which means two unrelated variables can end up neighbors in memory without either programmer intending it.
Multiple cores make CPU-bound work genuinely parallel (Part 1's real speedup) but introduce a brand-new question single-core programs never had to answer: what happens when two cores want the same memory at once?

C++ Multithreading: Why -O2 Deletes Your Loops and How std::atomic Fixes It

Ajinkya Singh — Mon, 17 Aug 2026 21:22:33 +0000

Recap

Part 1 raced an int and got a wrong number — 66666666 instead of
266666664, a lost-update bug you could point at and explain with three
lines of assembly. Wrong, but at least it looked like arithmetic gone
wrong. This article races something simpler — a bool — and the result
isn't a wrong number at all. It's the compiler quietly throwing your loop
away.

The setup: a flag, and a thread that waits on it

The pattern: one thread does some slow work and then flips a flag. Another
thread just spins, checking the flag, waiting for it to become true.

struct nonAtomic { bool ready = false; };
struct atomic    { std::atomic<bool> ready{ false }; };

template<typename T>
void runTest(const char* label) {
    T data;
    std::thread t1([&data]() {
        std::this_thread::sleep_for(std::chrono::milliseconds(7000));
        data.ready = true;
    });
    while (!data.ready) { }

    std::cout << label << " - Finished waiting for ready flag.\n";

    t1.join();
}

t1 sleeps 7 seconds, then sets ready = true. The main thread sits in
while (!data.ready) { }, doing nothing but re-checking, until it sees
that flip. Two versions of the same test — one with a plain bool, one
with std::atomic<bool> — run back to back:

void run() {
    std::thread t1(&CompilerHazard::runTest<nonAtomic>, this, "nonAtomic");
    std::thread t2(&CompilerHazard::runTest<atomic>, this, "atomic");

    t1.join();
    t2.join();
}

Nothing here looks dangerous. There's no +=, no lost update possible — a
bool can only be true or false, there's no "partial" value to race
over the way there was with cnt. Reasonable to assume this one's fine.

The problem: the compiler doesn't just get this wrong — it gets it wrong differently at every optimization level

Compile and run this exact code at three optimization levels and you get
three different stories:

At -O0 (no optimization): both versions work. Each one waits roughly
7 seconds, then prints its "finished waiting" line. Boring, correct, exactly
what you'd expect from reading the source.

At -O1: the atomic version still works correctly. The nonAtomic
version hangs forever. Not "takes longer" — never returns. t1 finishes
its sleep, sets ready = true, and the main thread's loop never notices.

At -O2: the atomic version is still correct, still ~7 seconds. The
nonAtomic version does something even stranger than hanging — it finishes
instantly, without ever actually waiting for t1's 7-second sleep at
all.

Same source code. Same hardware. Only the optimization flag changed, and
the plain-bool version gave three different, non-equivalent results
across three builds — one correct, one an infinite hang, one a nonsensical
instant "finish" that skipped the wait entirely.

Why: the compiler is allowed to assume this bug doesn't exist

while (!data.ready) { } re-checks ready on every loop iteration, hoping
to notice when another thread changes it. That's the intent.

But ready is a plain bool. One thread writes it, another thread reads
it, and nothing synchronizes the two. That's the exact definition of a data
race from Part 1 — and in C++, a data race is undefined behavior for the
entire program, not just that one variable.

Here's the part that trips people up: "undefined behavior" doesn't mean
"the compiler gives up." It means the compiler is legally allowed to
assume this situation never happens in a correct program. As far as the
optimizer can see, nothing in this function ever modifies ready once the
loop starts — so it treats ready as if it were constant for the rest of
the loop's life.

That one assumption plays out differently depending on how hard the
compiler is optimizing:

At -O1: the compiler reads ready once, stores it in a CPU register,
and just keeps re-checking that register forever — it never goes back out
to memory to look for t1's update. while (!data.ready) { } quietly
becomes while (true) { }. The check isn't wrong; it's just checking a
value that can never change.

At -O2: the compiler goes a step further. If no other thread is
allowed to touch ready, then the branch where this loop waits forever
for an external change is impossible — and impossible code is free to be
deleted. So it deletes the wait entirely and falls straight through, as if
the condition were already true.

Neither of these is a compiler bug. Both are legal optimizations, built on
a promise your code made by not synchronizing ready — and then broke.

`std::atomic` actually does two different jobs — don't blur them together

Here's where it's easy to get turned around: Part 1 fixed a race on cnt++.
This article is about a race on ready = true. Both get called "a data
race," both get fixed by std::atomic, and it's tempting to assume
std::atomic is one trick doing the same thing both times. It isn't. It's
two different guarantees, and each bug only needed one of them.

Job 1 — Atomicity: make a multi-step operation indivisible.
This is Part 1's problem. cnt += x is three instructions — read, modify,
write — and another thread can slip in between them. std::atomic<int>
fixes this by compiling the whole read-modify-write into one
hardware-locked instruction that cannot be interrupted:

; plain int:  cnt += x  -->  three separate, interruptible steps
mov  eax, [cnt]
add  eax, edx
mov  [cnt], eax

; std::atomic<int>:  cnt.fetch_add(x)  -->  one indivisible step
lock xadd [cnt], eax

The lock prefix tells the CPU: no other core may touch this memory
location until this single instruction finishes. There's no longer a
"between the read and the write" for another thread to land in, because
there's no longer a between at all.

Job 2 — Visibility: make a completed write reachable by other cores, in
order.
This is this article's problem. ready = true is already one
instruction — there's nothing to make indivisible, it's not being
interrupted mid-write. The bug isn't timing, it's that the compiler cached
a stale read, and separately, a real write can sit in a core's private
cache without a guaranteed moment of becoming visible elsewhere.
std::atomic<bool> fixes this by stopping the compiler from caching the
read, and by inserting the memory-ordering guarantees that make the write's
completion visible to other cores in a well-defined way.

Same keyword, two different repairs, applied depending on what's actually
wrong with the operation you wrapped in std::atomic<>. A single store
never needed Job 1 — it was already indivisible. A multi-step
read-modify-write always needs Job 1, and usually gets Job 2 along with it
for free, since std::atomic provides both by default.

Back to `ready` — this article only ever needed Job 2

Since ready = true was already one instruction, there was nothing here
for atomicity to fix. The actual gap was purely visibility:

Each core has its own cache. A write can sit there for a while before it's
visible to any other core. Without a synchronization point, there's no
promise about when — or even in what order — one core's writes show up to
another core's reads. std::atomic<bool> closes that gap: not by making a
single store "more atomic" (it already was one instruction), but by forcing
that store to become visible to other cores in a defined, guaranteed way,
and by stopping the compiler from short-circuiting the check with a cached
register value.

The full "why" behind cross-core visibility — cache lines, per-core caches,
the coherency protocol that keeps them honest — is what the Hardware Track
later in this series is for. For now, the one thing worth keeping straight:
"atomic" doesn't mean one fix for one kind of bug. It means "pick the
guarantee your specific operation is missing," and std::atomic<T> gives
you both, whichever one your code actually needed.

The fix: `std::atomic<bool>`

struct atomic {
    std::atomic<bool> ready{ false };
};

Same loop, same while (!data.ready) { }, only the type changes — and it
works at every optimization level, every time. For this specific bug, that's
entirely Job 2 doing the work: no more caching ready in a register across
iterations, and a guaranteed, ordered handoff of the write from one core to
the other.

std::atomic isn't "the fix for wrong numbers" or "the fix for hangs"
specifically — it's a toolbox with two tools in it, and which one your bug
needs depends on whether the operation you're protecting is multiple steps
(atomicity) or a single step whose visibility isn't guaranteed (ordering).
Wrong numbers, infinite hangs, and skipped checks are just different
costumes the same root cause — an unsynchronized shared variable — can wear,
depending on which tool was actually missing.

Why this matters more than the Part 1 bug did

Part 1's bug was loud, in a sense — the number was wrong, and if you had a
known-correct baseline to compare against (like we did), you'd catch it. This
bug is worse precisely because its behavior isn't even stable across
builds. Code that hangs at -O1 might look completely fine at -O0 in a
debug build during development, then hang in a release build in production.
Code that races at -O2 might not even wait for the condition it was
written to wait for, and silently proceed on data that isn't ready yet —
no crash, no hang, no obvious signal that anything went wrong at all.

"It worked when I tested it" was never proof of anything for a data race.
It's proof that this specific compiler, at this specific optimization
level, on this specific run, didn't happen to apply the optimization that
breaks you. Change any one of those three things and the same source code
can behave completely differently.

Takeaways

std::atomic does two distinct jobs, not one: atomicity (making a multi-step operation like cnt++ indivisible, via a locked instruction) and visibility (making a single-step write's completion reliably reach other cores, plus stopping the compiler from caching stale reads).
Part 1's cnt++ needed atomicity — it's three instructions with a gap to race in. This article's ready = true needed visibility — it's already one instruction, nothing to make more indivisible.
The exact failure mode (hang, instant skip, or correct-by-luck) can change across optimization levels for the same source file — there is no single "how it breaks," which is what makes this category of bug so hard to trust-by-testing.
Debug builds (-O0) hiding this class of bug entirely is a real and common trap — "no threading bugs in debug mode" tells you nothing about release mode.

Understanding C++ Threads: Data Races Explained Down to the Assembly

Ajinkya Singh — Sun, 16 Aug 2026 18:24:34 +0000

Notes & References

Sources this series builds on: Chili's (ChiliTomatoNoodle) multithreading
playlist, CMU 15-213 (Computer Systems: A Programmer's Perspective), and various CppCon talks.

Everything here is my own code, compiled and run — not just transcribed. These are my personal notes and references for future programming. LLMs were utilized for restructuring and sanity-checking the explanations.

The setup

One expensive, CPU-bound function. Every version in this article calls it —
only how many threads and how they share state changes.

void Thread::complexFunction(int& cnt) {
    const double PI_HALF = 1.5707963267948966;
    for (int i = 0; i < 100000000; ++i) {
        double angle = (double(i) / 100000000) * PI_HALF;
        cnt += (int)(std::sin(angle) * 2);
    }
}

Run it 4 times, back to back, on one thread:

void Thread::singleThread() {
    t.start();
    int cnt = 0;
    for (int i = 0; i < 4; i++) {
        complexFunction(cnt);
    }
    t.stop("Single Thread", true);
    std::cout << "cnt: " << cnt << "\n";
}

[Single Thread]: 3.04755 seconds
cnt: 266666664

That number, 266666664, is our source of truth for the rest of this
article. Every "correct" version below has to reproduce it.

The problem: the obvious multithreaded version is wrong

The natural next move: instead of calling complexFunction 4 times on one
thread, launch 4 threads and let them all run it at the same time, all
writing into the same cnt.

void Thread::simpleMultiThreadRaceCondition() {
    t.start();
    int cnt = 0;
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnt)));
    }
    for (auto& th : threads) {
        if (th.joinable()) {
            th.join();
        }
    }
    t.stop("Multi Thread Race Condition", true);
    std::cout << "cnt: " << cnt << "\n";
}

(Quick note on the mechanics here, since this is the first threaded code in
the series: std::ref(cnt) is required because std::thread copies its
arguments by default, and a plain int& parameter wouldn't survive that —
std::ref forces the real reference through. .join() in the loop after is
what blocks main until every worker is actually finished; skipping it and
letting threads go out of scope early is an instant crash, not a subtle bug.)

Run it:

[Multi Thread Race Condition]: 1.18949 seconds
cnt: 66666666

Faster, sure — that's expected, 4 cores instead of 1. But 66666666 is not
266666664. It's not close. It's suspiciously close to exactly
one quarter of the right answer — as if 3 out of every 4 threads'
work just vanished. The code compiled fine, ran fine, crashed nothing. It
just handed back the wrong number, silently.

Why: `cnt += x` is not one instruction

Here's the thing to unlearn: cnt += (int)(std::sin(angle) * 2); looks
like a single step in your source code. It is not one step for the CPU. At
-O0, it compiles to three separate instructions:

mov  eax, [cnt]      ; READ:   load cnt's current value into a register
add  eax, edx        ; MODIFY: add the new value to it
mov  [cnt], eax       ; WRITE:  store the result back into cnt

Read the value. Modify it. Write it back. Three distinct steps, three
distinct moments in time — and on a 4-core machine, all four threads are
doing this to the same address, at the same time, with no rule saying
one thread's three steps have to finish before another thread's three steps
begin.

Think of it like four people sharing one notebook with a running total
written in it, and no rule about taking turns:

Person A reads the notebook: "it says 100."
Before A writes anything back, Person B also reads it: "it also says 100."
A computes 100 + 3 = 103, and writes 103 into the notebook.
B, still working off the 100 it read a moment ago, computes 100 + 5 = 105, and writes 105 into the notebook — overwriting A's update completely.

Nobody gets an error. Nothing crashes. The notebook just silently ends up
wrong, and there's no record that A's update ever happened at all. This is
called a lost update, and it's what's known as a data race: two
threads touching the same memory, at least one of them writing, with no
coordination between them at all.

Do that millions of times a second, across four threads, for 100 million
iterations each, and you get a final number that isn't "slightly off" — it's
wrong in a way that happens to look patterned (close to 1/4 of the true
answer) purely because of how evenly the four threads' work happened to
collide on this run. Run it again and you'd likely get a different wrong
number. That's the part that should feel uncomfortable: the wrongness
itself isn't even consistent.

Solution attempt 1: lock the notebook — a mutex

If the problem is four threads reading and writing the same memory with no
coordination, the direct fix is to add coordination: force only one thread
at a time to be allowed inside the read-modify-write, no matter how many
threads are running.

void Thread::complexFunctionWithMutex(int& cnt) {
    const double PI_HALF = 1.5707963267948966;
    for (int i = 0; i < 100000000; ++i) {
            std::lock_guard<std::mutex> lock(mtx);
            double angle = (double(i) / 100000000) * PI_HALF;
            cnt += (int)(std::sin(angle) * 2);
    }
}

void Thread::simpleMultiThreadMutex() {
    t.start();
    int cnt = 0;
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunctionWithMutex, this, std::ref(cnt)));
    }
    for (auto& th : threads) {
        if (th.joinable()) {
            th.join();
        }
    }
    t.stop("Multi Thread Mutex", true);
    std::cout << "cnt: " << cnt << "\n";
}

std::lock_guard<std::mutex> lock(mtx); is the notebook's lock: whichever
thread grabs it first gets to finish its entire read-modify-write
uninterrupted, and every other thread has to wait its turn. No two threads
can be mid-update at the same time anymore — the exact gap that caused the
lost update is closed.

[Multi Thread Mutex]: 13.2063 seconds
cnt: 266666664

Correct — matches the single-threaded baseline exactly, every run. But look
at the time: 13.2 seconds, against a single-threaded baseline of 3.05
seconds. Using 4 cores made this over 4x slower than using one core and
no threading at all.

The reason is that we locked the mutex inside the loop, on every single
iteration — 400 million lock/unlock operations total, most of them spent
with three threads waiting idle for the fourth to finish its turn. We didn't
just add coordination — we added coordination so fine-grained that the four
threads are barely running in parallel at all; they're mostly taking turns,
plus paying the overhead of acquiring a lock 400 million times on top of that.

Correctness, fixed. Performance, worse than not threading in the first
place. That's a real trade-off, and it's hit constantly:
a mutex fixes the wrong answer problem, but a careless mutex can make your
"parallel" code slower than serial code.

Solution attempt 2: don't share the notebook at all

Here's the question worth sitting with: did these four threads ever actually
need to share cnt while they were working? Each thread's math is
completely independent of the other three — thread 2 never needs to know
what thread 1 computed mid-loop. The only reason we had a race at all is
that we chose to point all four threads at the same variable.

So — don't:

void Thread::simpleMultiThreadStoragePerThread() {
    t.start();
    int sum = 0;
    std::vector<int> cnts(4, 0);
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnts[i])));
    }

    for (auto& t : threads) {
        if (t.joinable()) {
            t.join();
        }
    }

    for (auto& cnt : cnts) {
        sum += cnt;
    }
    t.stop("Multi Thread Storage Per Thread", true);
    std::cout << "cnt: " << sum << "\n";
}

Give each thread its own slot (cnts[i]) instead of a shared one, let all
four run completely uninterrupted with no locking at all, and only add the
four totals together after every thread has joined — at which point no
thread is writing anymore, so there's nothing left to race.

[Multi Thread Storage Per Thread]: 1.12478 seconds
cnt: 266666664

Correct, and the fastest version by a clear margin — faster than the mutex
version by more than 10x, and even slightly faster than the naive (wrong)
race-condition version, because there's no locking overhead at all.

The full picture

Version	Time	`cnt`	Correct?
Single Thread	3.05s	266666664	✅ baseline
Race Condition (shared `cnt`, no lock)	1.19s	66666666	❌ lost updates
Mutex (shared `cnt`, locked every iteration)	13.2s	266666664	✅ but 4x slower than serial
Storage Per Thread (no sharing at all)	1.12s	266666664	✅ fastest overall

The lesson isn't "always use a mutex" or "always avoid shared state" — it's
that the fastest fix for a race is often to make the race impossible
rather than to police it. A mutex is the right tool when threads
genuinely need to coordinate over shared, mutable state. Here, they never
did — the sharing was the bug, not something that needed defending.

Takeaways

x += y is read-modify-write at the hardware level, not one atomic step — that gap between "one line of C++" and "three real instructions" is where every data race in this series starts.
A data race doesn't announce itself. It can produce a plausible-looking wrong number, and a different wrong number on the next run.
A mutex restores correctness by serializing access — but locking too finely (every iteration, in this case) can make multithreaded code slower than not threading at all.
If threads don't actually need to share mutable state, removing the sharing beats synchronizing it — no lock, no contention, no overhead.

Kubernetes Core Workloads: Everything You Need to Know

Ajinkya Singh — Fri, 03 Apr 2026 07:52:31 +0000

If you've made it past the basics of Kubernetes — you know what a cluster is, you've spun up a local environment with kind or minikube — the next wall you hit is understanding the workload objects. There are nine of them. They look similar in YAML. They all involve containers. But they each exist for a very different reason.

The Quick Reference: Which Workload Do You Need?

Before diving in, here's the decision tree you'll use every day:

Workload	Use When
Deployment	Running stateless apps, APIs, web servers, workers
StatefulSet	Databases, message brokers, anything needing stable identity
DaemonSet	Log agents, monitoring — one pod per node
Job	One-off batch tasks: migrations, reports, cleanup
CronJob	Scheduled recurring tasks
Pod (bare)	Almost never in production — use one of the above

Now let's understand why.

Pods: The Atomic Unit

Everything in Kubernetes ultimately runs as a Pod — the smallest deployable unit. You don't run containers directly; you run Pods that contain containers.

Think of a Pod as a tiny apartment that one or more containers share:

They share a single IP address
Containers within a Pod talk to each other via localhost
They can share volumes (disk storage)
They always land on the same node — never split

apiVersion: v1
kind: Pod
metadata:
  name: api-server-pod
spec:
  containers:
    - name: api
      image: mycompany/api:2.1.0
      resources:
        requests:
          memory: "256Mi"
          cpu: "250m"
        limits:
          memory: "512Mi"
          cpu: "1000m"
      livenessProbe:
        httpGet:
          path: /health
          port: 3000
        initialDelaySeconds: 15
        periodSeconds: 20
      readinessProbe:
        httpGet:
          path: /ready
          port: 3000
        initialDelaySeconds: 5
        periodSeconds: 10

The Most Important Thing About Pods

Pods are ephemeral. When a Pod dies, a new one is created with a completely new name and a completely new IP. Any code that hardcoded the old IP breaks. This is why Services exist (Phase 3) — but it's the mental model you need to carry through everything else.

Two Probes You Can't Skip

Liveness probe — "Is this container still alive?" If it fails, Kubernetes restarts the container. Use for crash detection and deadlocks.

Readiness probe — "Is this container ready for traffic?" If it fails, the pod is removed from the Service's endpoints — stops receiving traffic — but is not restarted. Use for startup warmup and temporary overload.

These two probes are doing different jobs. Never confuse them.

Pod Status Lifecycle

Pending → Running → Succeeded
              ↓
         Failed / CrashLoopBackOff / OOMKilled

When you see CrashLoopBackOff, run kubectl logs my-pod --previous — that's where your crash output lives.

Quick Debugging Playbook

# Pod stuck in Pending?
kubectl describe pod my-pod   # read the Events section
# "Insufficient CPU" → reduce requests or add nodes

# ImagePullBackOff?
kubectl describe pod my-pod   # wrong image name? typo? private registry?

# CrashLoopBackOff?
kubectl logs my-pod --previous   # read the crash

# OOMKilled?
kubectl describe pod my-pod   # see Last State → increase memory limit

# Running but not working?
kubectl exec -it my-pod -- /bin/bash   # investigate from inside

🧪 Practice — Pods

Lab 1: Your first pod

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: hello-nginx
  labels:
    app: hello
spec:
  containers:
    - name: nginx
      image: nginx:1.25
      ports:
        - containerPort: 80
      resources:
        requests:
          memory: "64Mi"
          cpu: "50m"
        limits:
          memory: "128Mi"
          cpu: "200m"
EOF

kubectl get pods --watch
kubectl describe pod hello-nginx    # read the Events section top to bottom
kubectl exec -it hello-nginx -- /bin/bash
# inside: curl localhost
kubectl port-forward pod/hello-nginx 8080:80 &
curl http://localhost:8080
kill %1

Lab 2: Multi-container pod — shared localhost

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: multi-pod
spec:
  containers:
    - name: app
      image: nginx:1.25
    - name: sidecar
      image: busybox
      command: ['sh', '-c', 'while true; do echo sidecar running; sleep 5; done']
EOF

kubectl logs multi-pod -c app
kubectl logs multi-pod -c sidecar
kubectl exec -it multi-pod -c sidecar -- /bin/sh
# inside: wget -q -O- localhost   ← hits the nginx container via shared IP

Lab 3: Trigger and observe CrashLoopBackOff

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: crasher
spec:
  containers:
    - name: app
      image: busybox
      command: ['sh', '-c', 'echo crashing; exit 1']
EOF

kubectl get pods --watch    # watch it go into CrashLoopBackOff
kubectl logs crasher --previous   # see the crash output

Cleanup

kubectl delete pod hello-nginx multi-pod crasher

Namespaces: Logical Isolation

A Namespace is a virtual cluster inside your real cluster. It's how you divide one Kubernetes cluster into isolated sections — by team, environment, or application.

Four namespaces exist by default:

default — where objects land if you don't specify
kube-system — Kubernetes system components (etcd, apiserver, coredns)
kube-public — publicly readable data (rarely used)
kube-node-lease — node heartbeat objects (internal, ignore)

What Namespaces Give You

Name reuse — you can have a webapp Deployment in both staging and production with no conflict.

RBAC scoping — give team A access only to their namespace.

Resource quotas — cap a namespace to 4 CPUs and 8Gi memory total.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    pods: "50"
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi

One Critical Misconception

Namespaces do not provide network isolation by default. A pod in dev can still reach a pod in prod if it knows the IP or DNS name. For actual network isolation, you need NetworkPolicies.

Cross-namespace DNS format: <service>.<namespace>.svc.cluster.local

🧪 Practice — Namespaces

Lab: Isolation, quotas, and the -n flag

# Create two environments
kubectl create namespace dev
kubectl create namespace prod

# Deploy the same app name in both — no conflict
kubectl create deployment webapp --image=nginx:1.25 -n dev
kubectl create deployment webapp --image=nginx:1.25 -n prod

kubectl get deployments -A    # see both side-by-side

# Apply a quota to prod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ResourceQuota
metadata:
  name: prod-quota
  namespace: prod
spec:
  hard:
    pods: "5"
    requests.cpu: "2"
    requests.memory: 4Gi
EOF

# Try to exceed it
kubectl scale deployment webapp --replicas=10 -n prod
kubectl get pods -n prod              # stops at quota limit
kubectl describe resourcequota prod-quota -n prod   # usage vs hard

# Set dev as your default namespace
kubectl config set-context --current --namespace=dev
kubectl get pods    # now implicitly reads from dev

# Switch back
kubectl config set-context --current --namespace=default

Cleanup

kubectl delete namespace dev prod

Labels, Selectors & Annotations

Labels are key-value pairs on any Kubernetes object. They're the connective tissue — how Services find Pods, how Deployments track their ReplicaSets, how you filter resources.

Annotations are also key-value pairs but for non-identifying metadata: descriptions, tool config, URLs. They cannot be used in selectors.

metadata:
  labels:
    app: api
    env: production
    version: "2.1.0"   # always quote values that look like numbers
  annotations:
    description: "Payments API server"
    prometheus.io/scrape: "true"

How a Service Uses Labels

apiVersion: v1
kind: Service
metadata:
  name: backend-service
spec:
  selector:
    app: api           # matches pods with app=api
    env: production    # AND env=production
  ports:
    - port: 80
      targetPort: 3000

The Service routes traffic to any pod with both labels. Remove a label from a pod and it's immediately removed from the Service's endpoints. Add it back and it rejoins. This is live — no restart required.

Set-Based Selectors

Beyond simple equality, you can use expressions:

selector:
  matchExpressions:
    - key: env
      operator: In
      values: [production, staging]
    - key: tier
      operator: NotIn
      values: [frontend]
    - key: app
      operator: Exists

One gotcha: always quote version numbers in YAML. version: 2.0 is parsed as a float, stored as "2" — breaking your selectors silently.

🧪 Practice — Labels & Selectors

Lab: Live label manipulation and Service routing

# Create three pods with different label combos
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: frontend-v1
  labels:
    app: frontend
    env: prod
    version: "1.0"
spec:
  containers:
    - name: nginx
      image: nginx:1.25
---
apiVersion: v1
kind: Pod
metadata:
  name: backend-v1
  labels:
    app: backend
    env: prod
    version: "1.0"
spec:
  containers:
    - name: nginx
      image: nginx:1.24
---
apiVersion: v1
kind: Pod
metadata:
  name: backend-staging
  labels:
    app: backend
    env: staging
    version: "2.0"
spec:
  containers:
    - name: nginx
      image: nginx:1.26
EOF

# Filter practice
kubectl get pods --show-labels
kubectl get pods -l app=backend
kubectl get pods -l app=backend,env=prod
kubectl get pods -l 'env in (prod,staging)'
kubectl get pods -l 'version notin (1.0)'

# Create a Service that selects app=backend + env=prod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: backend-svc
spec:
  selector:
    app: backend
    env: prod
  ports:
    - port: 80
      targetPort: 80
EOF

kubectl describe service backend-svc   # check Endpoints section

# Live label surgery — remove env from backend-v1
kubectl label pod backend-v1 env-
kubectl describe service backend-svc   # endpoint disappears immediately

# Add it back
kubectl label pod backend-v1 env=prod
kubectl describe service backend-svc   # endpoint returns

Cleanup

kubectl delete pod frontend-v1 backend-v1 backend-staging
kubectl delete service backend-svc

ReplicaSets: Self-Healing

A ReplicaSet ensures a specified number of identical Pods are always running.

spec.replicas: 3

Pod-2 crashes → ReplicaSet creates Pod-4 → back to 3
Rogue pod added → ReplicaSet deletes one → back to 3

The RS continuously counts pods matching its selector. Actual count ≠ desired → create or delete.

Why You Almost Never Create ReplicaSets Directly

ReplicaSets don't support rolling updates or rollbacks. That's exactly what Deployments add. In practice, you create a Deployment and it creates/manages a ReplicaSet for you.

You need to understand ReplicaSets because:

Deployments own them under the hood
Debugging often means inspecting the RS
It explains the self-healing behavior you see

The RS name format when created by a Deployment: <deployment-name>-<pod-template-hash>. That hash changes every time the pod template changes — which is exactly how rolling updates work.

🧪 Practice — ReplicaSets

Lab: Self-healing in action

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: demo-rs
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
EOF

kubectl get pods -l app=demo

# Open a watch in a second terminal: kubectl get pods --watch

# Kill one pod — RS heals immediately
kubectl delete pod $(kubectl get pods -l app=demo -o jsonpath='{.items[0].metadata.name}')
kubectl get pods -l app=demo    # new pod already created

# Scale up
kubectl scale rs demo-rs --replicas=5
kubectl get pods -l app=demo

# Try creating a stray pod with the same label
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: stray-pod
  labels:
    app: demo        # matches RS selector!
spec:
  containers:
    - name: nginx
      image: nginx:1.25
EOF

kubectl get pods -l app=demo    # RS sees 6, wants 5 → deletes one

Cleanup

kubectl delete rs demo-rs

Deployments: The Standard Way to Run Apps

A Deployment is what you use 90% of the time. It wraps a ReplicaSet and adds rolling updates, rollbacks, and version history.

You → Deployment → manages → ReplicaSets → manages → Pods
                      │
                      ├── RS v1 (old image)  → 0 pods
                      └── RS v2 (new image)  → 3 pods ✅

How a Rolling Update Actually Works

With maxSurge: 1, maxUnavailable: 0 and 3 replicas:

[v1][v1][v1]               # start

[v1][v1][v1][v2]           # spin up 1 new pod (surge)
[v1][v1][v2]               # v2 passes readiness → kill 1 v1

[v1][v1][v2][v2]           # spin up another v2
[v1][v2][v2]               # v2 passes → kill v1

[v1][v2][v2][v2]           # final new pod
[v2][v2][v2]               # all updated, zero downtime ✅

The readiness probe gates each step. Without it, Kubernetes can't tell when a new pod is actually ready.

Production Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0   # never go below desired count
  minReadySeconds: 10
  revisionHistoryLimit: 5
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: api
          image: mycompany/orders-api:3.0.1
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 5

The Commands You'll Use Daily

# Update image → triggers rolling update
kubectl set image deployment/my-app app=nginx:1.26

# Watch it happen
kubectl rollout status deployment/my-app

# Something went wrong?
kubectl rollout undo deployment/my-app

# Check history
kubectl rollout history deployment/my-app

# Restart all pods (rolling, with zero downtime)
kubectl rollout restart deployment/my-app

kubectl rollout undo is your emergency brake. Run it first, debug second.

🧪 Practice — Deployments

Lab: Full deployment lifecycle — deploy, update, break, rollback

# Deploy v1
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  annotations:
    kubernetes.io/change-cause: "initial deployment nginx 1.24"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
        - name: nginx
          image: nginx:1.24
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
EOF

kubectl get pods --watch    # Ctrl+C when all 3 Running
kubectl get rs              # see the RS created by the Deployment

# Rolling update to 1.25
kubectl set image deployment/webapp nginx=nginx:1.25
kubectl annotate deployment/webapp kubernetes.io/change-cause="upgrade to nginx 1.25" --overwrite
kubectl rollout status deployment/webapp
kubectl rollout history deployment/webapp   # see 2 revisions
kubectl get rs    # old RS now has 0 pods but still exists for rollback

# Simulate a bad update — nonexistent image
kubectl set image deployment/webapp nginx=nginx:NONEXISTENT
kubectl rollout status deployment/webapp    # hangs — new pods can't start
kubectl get pods    # old pods still serving! new pods in ErrImagePull

# Emergency rollback
kubectl rollout undo deployment/webapp
kubectl rollout status deployment/webapp    # watch recovery
kubectl get pods    # all healthy again

# Scale
kubectl scale deployment webapp --replicas=6
kubectl scale deployment webapp --replicas=2

Cleanup

kubectl delete deployment webapp

Imperative vs Declarative: The Mental Model

There are two ways to work with Kubernetes.

Imperative — tell Kubernetes what to do step by step:

kubectl create deployment web --image=nginx:1.25
kubectl scale deployment web --replicas=3
kubectl set image deployment/web nginx=nginx:1.26

Declarative — tell Kubernetes what you want, let it figure out how:

kubectl apply -f web.yaml
# edit the file
kubectl apply -f web.yaml   # kubernetes computes the diff

The key advantage of declarative: idempotency.

# Imperative — fails on second run
kubectl create deployment web --image=nginx
# Error: deployments.apps "web" already exists

# Declarative — safe to run forever
kubectl apply -f web.yaml
# deployment.apps/web created      (first run)
# deployment.apps/web unchanged    (no changes)
# deployment.apps/web configured   (file changed)

In production: always declarative. Store YAML in Git. CI/CD runs kubectl apply. This is GitOps — Git is the source of truth, the cluster always mirrors it.

Generate YAML Fast

kubectl create deployment web --image=nginx:1.25 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

--dry-run=client -o yaml generates the YAML locally without sending anything to the API server. Edit it, then apply. This is extremely useful in CKA/CKAD exams.

🧪 Practice — Imperative vs Declarative

Lab: Generate, edit, and apply

# Generate Deployment YAML without applying
kubectl create deployment web --image=nginx:1.25 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

cat deployment.yaml    # inspect what was generated

# Add resources block manually, then apply
kubectl apply -f deployment.yaml
kubectl apply -f deployment.yaml    # safe to run again — "unchanged"

# Edit the file — change replicas to 5
sed -i 's/replicas: 3/replicas: 5/' deployment.yaml
kubectl apply -f deployment.yaml    # "configured" — only diff applied
kubectl get deployment web

# Generate Pod and Service YAML for practice
kubectl run my-pod --image=busybox --dry-run=client -o yaml > pod.yaml
kubectl expose deployment web --port=80 --type=ClusterIP \
  --dry-run=client -o yaml > service.yaml

Cleanup

kubectl delete deployment web

DaemonSets: One Pod per Node

A DaemonSet ensures exactly one Pod runs on every node in the cluster. New node joins → pod auto-created. Node removed → pod garbage collected.

Node 1      Node 2      Node 3      Node 4 (new)
[log]       [log]       [log]       [log] ← auto-created

When to Use DaemonSets

Use Case	Examples
Log collection	Fluentd, Filebeat, Promtail
Monitoring agents	Prometheus node-exporter, Datadog agent
Network plugins	Calico, Cilium, kube-proxy
Security agents	Falco, Sysdig

kube-proxy itself is a DaemonSet. Verify: kubectl get ds -n kube-system

Running on Control-Plane Nodes

By default, DaemonSet pods won't run on control-plane nodes because of a taint: node-role.kubernetes.io/control-plane:NoSchedule. If you need them there, add a toleration:

spec:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

Update Strategies

RollingUpdate (default): automatically updates pods one node at a time
OnDelete: only updates pods when you manually delete them — useful for per-node maintenance windows

🧪 Practice — DaemonSets

Lab: Per-node coverage and system DaemonSets

# See existing DaemonSets in the cluster
kubectl get ds -n kube-system -o wide
# kube-proxy is a DaemonSet — one pod per node

# Create a simple node-monitoring DaemonSet
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-monitor
spec:
  selector:
    matchLabels:
      app: node-monitor
  template:
    metadata:
      labels:
        app: node-monitor
    spec:
      containers:
        - name: monitor
          image: busybox
          command:
            - sh
            - -c
            - while true; do echo "Node $(NODE_NAME) alive at $(date)"; sleep 10; done
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests:
              memory: "20Mi"
              cpu: "10m"
EOF

# Verify one pod per worker node
kubectl get pods -l app=node-monitor -o wide

# Check logs from one
POD=$(kubectl get pods -l app=node-monitor -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD

Cleanup

kubectl delete ds node-monitor

Jobs & CronJobs: Batch and Scheduled Work

Job — runs pods to completion. Once finished successfully, stops. For one-off batch tasks.

CronJob — runs a Job on a schedule (cron syntax). For recurring tasks.

Deployment:  "Run this forever"
Job:         "Run this ONCE until it succeeds"
CronJob:     "Run this Job every day at 2am"

Job YAML

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  backoffLimit: 3              # retry up to 3 times
  activeDeadlineSeconds: 300   # kill after 5 minutes regardless
  ttlSecondsAfterFinished: 60  # auto-delete 60s after completion
  template:
    spec:
      restartPolicy: OnFailure  # REQUIRED — Always is invalid for Jobs
      containers:
        - name: migration
          image: mycompany/api:3.0.1
          command: ["node", "scripts/migrate.js"]

The restartPolicy: OnFailure (or Never) is mandatory — Jobs cannot use Always because they'd never complete.

Set ttlSecondsAfterFinished or finished Jobs pile up in your cluster forever.

CronJob YAML

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"        # every day at 2:00 AM
  concurrencyPolicy: Forbid    # skip if previous run still running
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 2
      ttlSecondsAfterFinished: 300
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: reporter
              image: mycompany/reporter:1.0
              command: ["python", "generate_report.py"]

Cron Schedule Quick Reference

0 2 * * *       Every day at 2:00 AM
*/5 * * * *     Every 5 minutes
0 9 * * 1       Every Monday at 9:00 AM
0 0 1 * *       1st of every month at midnight
@daily          Shortcut for 0 0 * * *
@hourly         Shortcut for 0 * * * *

Trigger a CronJob Manually

kubectl create job --from=cronjob/nightly-report manual-run-01

This creates a Job immediately using the CronJob's template, without affecting scheduled runs.

🧪 Practice — Jobs & CronJobs

Lab 1: Simple one-off Job

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: hello-job
spec:
  backoffLimit: 2
  ttlSecondsAfterFinished: 60
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: hello
          image: busybox
          command: ['sh', '-c', 'echo "Job running at $(date)"; sleep 5; echo Done!']
EOF

kubectl get pods --watch    # watch pod go: Pending → Running → Completed
kubectl logs -l job-name=hello-job
kubectl get job hello-job   # COMPLETIONS should show 1/1

Lab 2: CronJob that runs every minute

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: minute-job
spec:
  schedule: "* * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 60
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: echo
              image: busybox
              command: ['sh', '-c', 'echo "Scheduled run at $(date)"']
EOF

# Wait 1-2 minutes
kubectl get jobs
kubectl get cronjob minute-job

# Trigger it manually right now
kubectl create job --from=cronjob/minute-job manual-run-01
kubectl logs -l job-name=manual-run-01

# Suspend future runs without deleting
kubectl patch cronjob minute-job -p '{"spec":{"suspend":true}}'
kubectl get cronjob minute-job    # SUSPEND should show True

Cleanup

kubectl delete cronjob minute-job

StatefulSets: Stateful Applications

A StatefulSet is like a Deployment, but for applications that need three guarantees:

Stable, predictable names — pods are always pod-0, pod-1, pod-2
Stable storage — each pod gets its own PVC that sticks with it across restarts
Ordered startup/shutdown — pods start in order (0→1→2) and stop in reverse

DEPLOYMENT:              STATEFULSET:
my-app-abc12 (random)    mysql-0  (always 0)
my-app-def34 (random)    mysql-1  (always 1)
my-app-ghi56 (random)    mysql-2  (always 2)

Pod dies → new name      Pod dies → comes back as mysql-1
Pod dies → new IP        Pod dies → same storage re-attached
Starts in any order      Starts 0 first, then 1, then 2

The Headless Service Requirement

StatefulSets require a headless Service (clusterIP: None) to provide stable DNS names per pod:

mysql-0.mysql-svc.default.svc.cluster.local → 10.244.1.5
mysql-1.mysql-svc.default.svc.cluster.local → 10.244.2.8

A normal Service load-balances across pods. A headless Service returns individual DNS records for each pod — so applications can say "connect to the primary at mysql-0.mysql-svc" and mean it.

StatefulSet YAML

# Headless Service — create this FIRST
apiVersion: v1
kind: Service
metadata:
  name: mysql-svc
spec:
  clusterIP: None    # ← this makes it headless
  selector:
    app: mysql
  ports:
    - port: 3306
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-svc    # links to headless service above
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:     # ← key difference from Deployment
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

volumeClaimTemplates creates individual PVCs per pod: data-mysql-0, data-mysql-1, data-mysql-2.

The PVC Retention Gotcha

When you delete a StatefulSet, the PVCs are not deleted. This is intentional — data is precious. But it means they stay around until you clean them up manually:

kubectl delete sts mysql
kubectl delete pvc -l app=mysql   # must do this separately

Canary Updates with partition

The partition field in rollingUpdate is powerful for staged rollouts:

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    partition: 2    # only update pods with ordinal >= 2

Set partition: 2 to update only mysql-2 first. Test it. Lower to 1. Lower to 0 to complete. This is native canary deployment for StatefulSets.

🧪 Practice — StatefulSets

Lab: Stable identity, persistent storage, and ordered startup

# Create headless service + StatefulSet
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: web-svc
spec:
  clusterIP: None
  selector:
    app: web
  ports:
    - port: 80
      name: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: web-svc
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: www
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
EOF

# Watch ordered startup — web-0 first, then web-1, then web-2
kubectl get pods --watch

# Confirm stable names
kubectl get pods -l app=web   # always web-0, web-1, web-2

# See the per-pod PVCs created
kubectl get pvc   # www-web-0, www-web-1, www-web-2

# Write data to web-0's volume
kubectl exec web-0 -- sh -c 'echo "pod-0 data" > /usr/share/nginx/html/index.html'

# Delete web-0 — watch it come back with the SAME name
kubectl delete pod web-0
kubectl get pods --watch    # web-0 comes back, not a random name

# Data persists — PVC was reattached
kubectl exec web-0 -- cat /usr/share/nginx/html/index.html   # still "pod-0 data"

# Reach a specific pod by stable DNS from another pod
kubectl run tmp --image=busybox --rm -it --restart=Never -- \
  wget -q -O- web-0.web-svc.default.svc.cluster.local

Cleanup — note PVCs must be deleted separately

kubectl delete sts web
kubectl delete svc web-svc
kubectl delete pvc -l app=web    # PVCs are NOT auto-deleted

The Master Cheat Sheet

# LIST
kubectl get pods / deploy / rs / ds / sts / jobs / cronjobs

# USEFUL FLAGS
kubectl get pods -o wide              # +IP +NODE
kubectl get pods --show-labels        # show label columns
kubectl get pods -l app=my-app        # filter by label
kubectl get pods -A                   # all namespaces

# INSPECT (always read the Events section)
kubectl describe pod/deploy/sts <n>

# LOGS
kubectl logs <pod>
kubectl logs <pod> --previous         # after a crash
kubectl logs <pod> -c <container>     # multi-container pod

# SHELL
kubectl exec -it <pod> -- /bin/bash

# DEPLOYMENTS
kubectl scale deploy <n> --replicas=5
kubectl set image deploy/<n> <container>=<image>:tag
kubectl rollout status deploy/<n>
kubectl rollout undo deploy/<n>
kubectl rollout restart deploy/<n>

# JOBS
kubectl create job test --from=cronjob/<n>   # manual trigger
kubectl patch cronjob <n> -p '{"spec":{"suspend":true}}'

# GENERATE YAML BOILERPLATE
kubectl create deployment web --image=nginx:1.25 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

API Versions Quick Reference

Workload	apiVersion	kind
Pod	`v1`	`Pod`
Namespace	`v1`	`Namespace`
ReplicaSet	`apps/v1`	`ReplicaSet`
Deployment	`apps/v1`	`Deployment`
DaemonSet	`apps/v1`	`DaemonSet`
StatefulSet	`apps/v1`	`StatefulSet`
Job	`batch/v1`	`Job`
CronJob	`batch/v1`	`CronJob`

Restart Policy Rules

Policy	Use In	Behaviour
`Always`	Deployments, DaemonSets	Restart on any exit — default
`OnFailure`	Jobs	Restart only on non-zero exit
`Never`	Jobs	Never restart — new pod on each retry

🧪 Final Practice — End-to-End Scenario

This lab ties everything together. You'll deploy a full application stack using every concept from this article.

Scenario: Deploy a web application with a background worker, a nightly cleanup job, and per-node logging.

# Step 1: Create a dedicated namespace
kubectl create namespace myapp

# Step 2: Deploy the web app (Deployment)
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: myapp
  labels:
    app: web
    tier: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
        tier: frontend
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          resources:
            requests: { memory: "64Mi", cpu: "50m" }
            limits:   { memory: "128Mi", cpu: "200m" }
          readinessProbe:
            httpGet: { path: /, port: 80 }
            initialDelaySeconds: 3
EOF

# Step 3: Deploy a background worker (Deployment, different labels)
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
  namespace: myapp
  labels:
    app: worker
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: worker
  template:
    metadata:
      labels:
        app: worker
        tier: backend
    spec:
      containers:
        - name: worker
          image: busybox
          command: ['sh', '-c', 'while true; do echo "worker processing..."; sleep 10; done']
          resources:
            requests: { memory: "32Mi", cpu: "25m" }
EOF

# Step 4: Per-node log collector (DaemonSet)
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: myapp
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      containers:
        - name: collector
          image: busybox
          command: ['sh', '-c', 'while true; do echo "collecting logs from $(NODE_NAME)"; sleep 15; done']
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests: { memory: "20Mi", cpu: "10m" }
EOF

# Step 5: Nightly cleanup Job (CronJob)
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cleanup
  namespace: myapp
spec:
  schedule: "0 3 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 120
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: cleanup
              image: busybox
              command: ['sh', '-c', 'echo "cleaning up old data..."; sleep 3; echo done']
EOF

# Inspect the whole namespace
kubectl get all -n myapp

# Check labels
kubectl get pods -n myapp --show-labels

# Filter by tier
kubectl get pods -n myapp -l tier=frontend
kubectl get pods -n myapp -l tier=backend

# Trigger cleanup job manually
kubectl create job --from=cronjob/cleanup manual-cleanup-01 -n myapp
kubectl logs -l job-name=manual-cleanup-01 -n myapp

# Simulate rolling update on web
kubectl set image deployment/web nginx=nginx:1.26 -n myapp
kubectl rollout status deployment/web -n myapp

# Scale worker up
kubectl scale deployment worker --replicas=3 -n myapp
kubectl get pods -n myapp

Cleanup

kubectl delete namespace myapp    # deletes everything inside it

Kubernetes Foundations — Architecture, Components & kubectl

Ajinkya Singh — Sun, 22 Mar 2026 19:05:21 +0000

Why Kubernetes Exists
Cluster Architecture
Control Plane
Worker Node Components
kubectl
Local Cluster Setup
Cheat Sheet

Why Kubernetes Exists

Before Kubernetes, running containers in production meant: SSH into servers, start and stop containers by hand, write fragile shell scripts for restarts, and scale by spinning up machines one by one.

Kubernetes is a container orchestration system. It solves five problems that every team hits at scale.

🔴 Problem 1 — Single Point of Failure

Without Kubernetes:
  [App] ──► [Server A] ──✕──► App is DOWN until someone manually fixes it

With Kubernetes:
             ┌──► [Node A] ──✕── dies
  [App] ─────┤──► [Node B]  ✓  still running
             └──► [Node C]  ✓  still running
                            K8s auto-reschedules pod from A → B or C

Kubernetes detects the dead node and automatically reschedules all pods to healthy ones. No pager. No SSH.

📈 Problem 2 — Manual Scaling

Without Kubernetes:
  Traffic spike → someone notices → SSH → docker run ×10 → 20 minutes wasted

With Kubernetes:
  Traffic spike → HPA detects high CPU → pods spin up automatically → seconds, no human

The Horizontal Pod Autoscaler watches metrics and adds or removes pods automatically.

🚀 Problem 3 — Deployment Downtime

Without Kubernetes:
  Deploy v2:   stop v1 ──► [gap — users see errors] ──► start v2

With Kubernetes (rolling update):
  [v1][v1][v1]
       ↓
  [v2][v1][v1]   ← v2 started and health-checked first
       ↓
  [v2][v2][v1]
       ↓
  [v2][v2][v2]   ← v1 fully gone. Zero downtime.

New pods are health-checked before old ones are removed. Users never see an error page.

🔑 Problem 4 — Config & Secrets Chaos

Without Kubernetes:
  DB_PASSWORD="hunter2"  ← hardcoded in source code
  Different .env files per environment ← drift guaranteed

With Kubernetes:
  ConfigMaps + Secrets   ← config lives outside the image
  Same binary            ← works in dev / staging / prod

Config is decoupled from the container image entirely.

🌐 Problem 5 — Service Discovery Breakage

Without Kubernetes:
  Service A ──hardcoded──► IP 10.0.0.45 (Service B)
  Service B restarts → new IP → Service A breaks

With Kubernetes:
  Service A ──DNS──► "payment-service"
  Kubernetes routes to the right pods — IPs are irrelevant

Pods are ephemeral and get new IPs on restart. Kubernetes Services provide stable DNS names that always resolve correctly.

Common Misconceptions

Misconception	Reality
Kubernetes replaces Docker	No — K8s orchestrates containers; Docker/containerd runs them
Only for large companies	Works equally well for a 2-person startup
Cloud-only	Runs on laptops, on-prem hardware, and cloud equally
Replaces CI/CD pipelines	K8s is the deployment target, not the pipeline

Cluster Architecture

A Kubernetes cluster has two types of machines:

Control Plane — the brain. Makes all scheduling and reconciliation decisions.
Worker Nodes — the muscle. Runs your actual application containers.

All communication goes through the control plane. You never interact with worker nodes directly.

┌──────────────────────────────────────────────────────────────────────┐
│                        KUBERNETES CLUSTER                            │
│                                                                      │
│   You (kubectl)                                                      │
│        │  HTTPS                                                      │
│        ▼                                                             │
│   ┌──────────────────────────────────────────────────────────────┐  │
│   │                     CONTROL PLANE                            │  │
│   │                                                              │  │
│   │   kube-apiserver  ◄──────────────►  etcd                    │  │
│   │   (front door · port 6443)          (cluster database)      │  │
│   │                                                              │  │
│   │   kube-scheduler                  controller-manager        │  │
│   │   (pod placement)                 (reconciliation loops)    │  │
│   └───────────────┬──────────────────────────┬────────────────  ┘  │
│                   │                          │                      │
│      ┌────────────▼──────┐  ┌────────────────▼──┐  ┌────────────┐  │
│      │   Worker Node 1   │  │   Worker Node 2   │  │  Worker 3  │  │
│      │  Pod  Pod         │  │   Pod             │  │  Pod  Pod  │  │
│      │  kubelet          │  │   kubelet         │  │  kubelet   │  │
│      │  kube-proxy       │  │   kube-proxy      │  │  kube-proxy│  │
│      │  containerd       │  │   containerd      │  │  containerd│  │
│      └───────────────────┘  └───────────────────┘  └────────────┘  │
└──────────────────────────────────────────────────────────────────────┘

▶ Full flow: kubectl create deployment → running container

1.   kubectl          reads ~/.kube/config → finds API server URL
2.   kube-apiserver   authenticates, RBAC check, validates request
3.   kube-apiserver   writes Deployment object to etcd
4.   controller-mgr   detects new Deployment → creates ReplicaSet → unscheduled Pod
5.   kube-scheduler   filters + scores nodes → binds pod to best node
6.   kubelet          sees assignment → calls containerd via CRI
7.   containerd       pulls image → calls runc → starts container
8.   kubelet          reports "Running" to API server
9.   etcd             pod status updated
10.  kubectl get pods → my-app  1/1  Running ✓

Control Plane

kube-apiserver

The single entry point for everything — kubectl, CI/CD pipelines, dashboards, and internal components. Validates requests, enforces RBAC, persists state to etcd. Port 6443 over HTTPS.

▶ The 4-gate request pipeline

Incoming request
       │
       ▼
┌─ GATE 1: AUTHENTICATION ─────────────────────────────────────────┐
│  "Who are you?"                                                  │
│  TLS client certificates · Bearer tokens · OIDC                 │
│  Failure → 401 Unauthorized                                      │
└──────────────────────────────────────┬───────────────────────────┘
                                       │
                                       ▼
┌─ GATE 2: AUTHORIZATION (RBAC) ───────────────────────────────────┐
│  "Are you allowed to do this?"                                   │
│  Checks Roles and ClusterRoles bound to the user                 │
│  Failure → 403 Forbidden                                         │
└──────────────────────────────────────┬───────────────────────────┘
                                       │
                                       ▼
┌─ GATE 3: ADMISSION CONTROL ──────────────────────────────────────┐
│  Mutating webhooks   → can modify the request                    │
│  Validating webhooks → can reject the request                    │
│  Failure → 400 / 422                                             │
└──────────────────────────────────────┬───────────────────────────┘
                                       │
                                       ▼
┌─ GATE 4: PERSIST TO etcd ────────────────────────────────────────┐
│  Object written → 201 Created returned ✓                         │
└──────────────────────────────────────────────────────────────────┘

Controllers and kubectl --watch use long-lived HTTP watch streams, not polling:

# GET /api/v1/namespaces/default/pods?watch=true
# Server streams:
# ADDED    my-pod   Pending
# MODIFIED my-pod   Running
# DELETED  my-pod

etcd

Distributed key-value store. The single source of truth for all cluster state. Losing etcd without a backup = losing all cluster configuration.

/registry/pods/default/my-pod         →  { pod spec }
/registry/deployments/default/my-app  →  { deployment spec }
/registry/services/default/my-svc     →  { service config }

etcd uses Raft consensus — a write commits only when a majority quorum agrees.

▶ Raft quorum explained

3-node etcd cluster:

  Leader ──heartbeat──► Follower 1
         ──heartbeat──► Follower 2

Write committed when majority (2 of 3) agree ✓
Leader + one Follower offline → 1 of 3 → no majority → writes halt ✗

Cluster Size	Quorum	Failures Tolerated
1	1	0
3	2	1 ← production minimum
5	3	2 ← recommended
7	4	3

Always use an odd number. Even numbers risk a 50/50 split → no quorum → all writes halt.

kube-scheduler

Decides which node a new pod runs on. Does not start the pod — only decides placement.

▶ How the scheduler picks a node

New pod (unscheduled)
       │
       ▼
PHASE 1 — FILTER: "Which nodes can run this pod?"
  Node 1: 90% CPU → skip
  Node 2: 20% CPU → feasible ✓
  Node 3: 55% CPU → feasible ✓

PHASE 2 — SCORE: "Which is the best node?"
  Node 2: score 85  (less loaded, image already cached)
  Node 3: score 62

PHASE 3 — BIND:
  Pod assigned to Node 2 → kubelet starts the container

Diagnosing a pod stuck in Pending:

kubectl describe pod my-pod
# Events:
# Warning  FailedScheduling  0/3 nodes are available:
#   3 Insufficient memory.

controller-manager

Runs all built-in controllers in a single binary. Each one watches for drift between desired and actual state and corrects it — forever.

while True:
    desired = api_server.get_desired_state()
    actual  = api_server.get_actual_state()
    if desired != actual:
        take_corrective_action()   # create / delete / update
    sleep(brief_interval)

▶ Built-in controllers

Controller	Responsibility
Deployment Controller	Manages ReplicaSets to maintain desired replicas
ReplicaSet Controller	Ensures the correct number of Pods exist
Node Controller	Detects node failures, evicts pods from dead nodes
Job Controller	Runs pods to completion for batch workloads
CronJob Controller	Creates Jobs on a cron schedule
Endpoint Controller	Updates Service endpoints when backing pods change
Namespace Controller	Cleans up resources when a namespace is deleted

Production note: Run 3 or 5 control plane nodes for HA. Back up etcd on a schedule. Never run user workloads on control plane nodes.

Worker Node Components

kubelet

Runs on every worker node as a systemd service — not a Kubernetes pod. If it crashes, systemd restarts it.

▶ What kubelet manages

Responsibility	Detail
PodSpecs	Downloads pod definitions from the API server
Container lifecycle	Start, stop, restart via the container runtime
Health checks	Runs liveness, readiness, and startup probes
Resource reporting	Reports CPU/memory usage to the API server
Volume mounting	Mounts ConfigMaps, Secrets, and PVCs into containers
Static pods	Reads YAML from `/etc/kubernetes/manifests/` directly

▶ Static pods — how the control plane bootstraps itself

Static pods allow kubelet to start containers without a running API server, by reading YAML files from disk. This is how K8s starts itself from scratch:

/etc/kubernetes/manifests/
  etcd.yaml
  kube-apiserver.yaml
  kube-controller-manager.yaml
  kube-scheduler.yaml

Bootstrap order:
  systemd starts kubelet
    → kubelet reads manifests → starts etcd
    → starts kube-apiserver
    → everything else connects normally

# Inspect in a kind cluster:
docker exec kind-control-plane ls /etc/kubernetes/manifests/

kube-proxy

Programs iptables or IPVS rules into the Linux kernel for Service routing. Not in the packet data path at runtime — it writes the rules and steps aside.

Client → GET http://my-service:80
       ↓
Linux kernel intercepts (rules written by kube-proxy)
       ├── 33% ──► Pod 10.244.1.5 (Node 1)
       ├── 33% ──► Pod 10.244.2.8 (Node 2)
       └── 34% ──► Pod 10.244.3.2 (Node 3)

▶ Proxy modes comparison

Mode	Mechanism	Lookup	Use Case
`iptables` (default)	Linux iptables chains	O(n)	< ~1,000 Services
`ipvs`	Kernel hash table	O(1)	Large clusters
eBPF (Cilium)	Bypasses iptables entirely	Best-in-class	Performance-critical

containerd

The default container runtime. kubelet calls it via the CRI (Container Runtime Interface).

kubelet → (gRPC / CRI) → containerd → (OCI) → runc → container

▶ The "Docker was removed" myth — clarified

What was removed in K8s 1.24: dockershim (an adapter between kubelet and Docker's API).

What was NOT removed:
  ✓  Docker images and Dockerfiles → still work perfectly
  ✓  docker build                  → still valid locally

Before K8s 1.24:  kubelet → dockershim → Docker → containerd → runc
After  K8s 1.24:  kubelet → containerd → runc

Docker was a middleman. containerd — which Docker itself uses internally — always did the actual work.

Runtime	Used By	Notes
containerd	GKE, EKS, AKS, kind	Default — lightweight, fast
CRI-O	OpenShift, bare-metal	Built specifically for Kubernetes

kubectl

kubectl reads ~/.kube/config to find your cluster's API server URL and credentials, then talks to kube-apiserver over HTTPS.

Managing Multiple Clusters

kubectl config view
kubectl config get-contexts
kubectl config use-context kind-k8s-lab
kubectl config current-context
kubectl config set-context --current --namespace=my-team
kubectl get pods --context=production-cluster

Command Structure

kubectl  [verb]    [resource]   [name]      [flags]
         get       pods         my-pod      -n production
         describe  deployment   my-dep
         delete    service      my-svc
         apply                  -f app.yaml
         exec      -it my-pod   --          /bin/bash
         logs      my-pod                   -f --previous

▶ Getting information

kubectl get pods
kubectl get pods -n kube-system        # specific namespace
kubectl get pods -A                    # all namespaces
kubectl get pods -o wide               # with IP and NODE columns
kubectl get pods -o yaml               # full YAML spec
kubectl get pods --watch               # live stream
kubectl get pods -l app=my-app         # filter by label
kubectl get all -n my-namespace

kubectl describe pod my-pod            # full details + Events ← debug here
kubectl describe node my-node
kubectl describe deployment my-dep

> Always read the Events section in kubectl describe — it is the primary debugging signal in Kubernetes.

▶ Creating and updating resources

kubectl apply -f app.yaml              # create or update (idempotent) ← always prefer this
kubectl apply -f ./manifests/
kubectl create deployment my-app --image=nginx
kubectl create namespace my-team

kubectl apply is idempotent — creates if absent, updates if present. kubectl create fails if the resource already exists. Use apply in all scripts and pipelines.

▶ Deleting resources

kubectl delete pod my-pod
kubectl delete -f app.yaml
kubectl delete deployment my-app                        # removes ReplicaSet and pods too
kubectl delete pod my-pod --force --grace-period=0      # immediate (stuck pods)

▶ Logs

kubectl logs my-pod
kubectl logs my-pod -f                      # stream live
kubectl logs my-pod --previous              # logs before last restart ← use after crashes
kubectl logs my-pod -c my-container         # specific container
kubectl logs my-pod --tail=100
kubectl logs my-pod --since=1h
kubectl logs -l app=my-app                  # all matching pods

▶ Exec into a pod

kubectl exec -it my-pod -- /bin/bash
kubectl exec -it my-pod -- /bin/sh          # if bash is unavailable
kubectl exec my-pod -- ls /app
kubectl exec -it my-pod -c sidecar -- /bin/sh

# Ephemeral debug pod — auto-deleted on exit
kubectl run debug --image=busybox --rm -it --restart=Never -- /bin/sh

▶ Port forwarding

kubectl port-forward pod/my-pod 8080:80
kubectl port-forward service/my-svc 8080:80
kubectl port-forward deployment/my-app 8080:3000

▶ Scaling and rollouts

kubectl scale deployment my-app --replicas=5
kubectl set image deployment/my-app nginx=nginx:1.25
kubectl rollout status deployment/my-app
kubectl rollout history deployment/my-app
kubectl rollout undo deployment/my-app
kubectl rollout undo deployment/my-app --to-revision=3

▶ JSONPath — extract specific fields

kubectl get pod my-pod -o jsonpath='{.status.podIP}'
kubectl get pods -o jsonpath='{.items[*].metadata.name}'

# All pods: namespace, name, IP
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'

Common Mistakes

Mistake	Correct Approach
Using `kubectl create` in scripts	Use `kubectl apply` — idempotent
Forgetting `-n <namespace>`	Use `-A` or set a default namespace in the context
Deleting a pod and expecting it to stay gone	Delete the Deployment — the ReplicaSet recreates pods
Skipping the Events section in `describe`	Events are the primary debugging signal
Not using `--previous` after a crash	`kubectl logs my-pod --previous` shows pre-crash logs

Local Cluster Setup

▶ Tool comparison

Tool	Nodes	Use Case
kind (recommended)	Multi-node	CI/CD, multi-node testing, closest to production
minikube	Single-node	Quick experiments, rich addons

Install

# macOS
brew install kubectl kind

# Linux — kubectl
curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# Linux — kind
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
chmod +x kind && sudo mv kind /usr/local/bin/

Create a Multi-Node Cluster

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: k8s-lab
nodes:
  - role: control-plane
  - role: worker
  - role: worker

kind create cluster --config kind-config.yaml

kubectl get nodes
# k8s-lab-control-plane  Ready  control-plane
# k8s-lab-worker         Ready  <none>
# k8s-lab-worker2        Ready  <none>

Verify Health

kubectl get nodes                   # all nodes → Ready
kubectl get pods -n kube-system     # all system pods → Running
kubectl cluster-info

▶ First application lab

# Deploy
kubectl create deployment my-nginx --image=nginx:1.25 --replicas=2
kubectl get pods -o wide

# Expose
kubectl expose deployment my-nginx --port=80 --name=my-nginx-service
kubectl describe service my-nginx-service

# Access
kubectl port-forward service/my-nginx-service 8080:80 &amp;
curl http://localhost:8080

# Scale
kubectl scale deployment my-nginx --replicas=4

# Rolling update
kubectl set image deployment/my-nginx nginx=nginx:1.26
kubectl rollout status deployment/my-nginx

# Rollback
kubectl rollout undo deployment/my-nginx

# Clean up
kill %1
kubectl delete deployment my-nginx
kubectl delete service my-nginx-service

kind Management

kind get clusters
kind create cluster --config kind-config.yaml
kind delete cluster --name k8s-lab
kind load docker-image my-app:latest --name k8s-lab
kubectl config use-context kind-k8s-lab

Cheat Sheet

▶ Architecture & key concepts

Control Plane:   kube-apiserver · etcd · kube-scheduler · controller-manager
Worker Nodes:    kubelet · kube-proxy · containerd
Full flow:       kubectl → apiserver → etcd/scheduler → kubelet → container

Reconciliation:  desired ≠ actual → controller fixes it → repeat forever
Static pods:     /etc/kubernetes/manifests/ → kubelet reads directly
kubeconfig:      ~/.kube/config → cluster URLs + credentials
etcd quorum:     always odd — 3 or 5 in production
Scheduler:       filter → score → bind to winner
kubelet:         systemd service (not a pod); bootstraps the control plane

▶ kubectl quick reference

# Get
kubectl get pods / svc / deploy / nodes
kubectl get pods -A                      # all namespaces
kubectl get pods -o wide                 # +IP +NODE
kubectl get pod  -o yaml             # full spec
kubectl get pods --watch                 # live stream
kubectl describe pod                # details + Events ← debug here
kubectl explain pod.spec.containers      # built-in docs

# Apply / Delete
kubectl apply -f file.yaml
kubectl delete -f file.yaml
kubectl delete pod  --force

# Logs
kubectl logs  -f
kubectl logs  --previous            # after crash ← important
kubectl logs  -c 
kubectl logs -l app=my-app

# Exec / Port-forward
kubectl exec -it  -- /bin/bash
kubectl port-forward svc/ 8080:80

# Scale / Rollout
kubectl scale deployment  --replicas=5
kubectl set image deployment/ =<img>:tag
kubectl rollout status deployment/
kubectl rollout history deployment/
kubectl rollout undo deployment/

# Context
kubectl config get-contexts
kubectl config use-context 
kubectl config set-context --current --namespace=

▶ kind quick reference

kind create cluster --config kind-config.yaml
kind get clusters
kind load docker-image  --name 
kind delete cluster --name

🤖 AI-generated · Personal reference only · Not original content"

📚 Complete Java Microservices Guide: Dropwizard & Ecosystem

Ajinkya Singh — Thu, 26 Feb 2026 17:39:46 +0000

A Step-by-Step Beginner-to-Intermediate Learning Path

How to use this guide: Read chapter by chapter. Every concept builds on the previous one. Code examples are practical and runnable. Chapters marked 🔲 are TODO — topics to study next.

📋 Table of Contents

#	Chapter	Status
1	What is Dropwizard & Why Use It?	✅
2	Project Setup & Fat JAR	✅
3	Configuration with YAML	✅
4	Building REST Resources (JAX-RS)	✅
5	Bean Validation (Hibernate Validator)	✅
6	Jackson – JSON Serialization	✅
7	Database Access with Hibernate/JDBI	✅
8	Liquibase – Database Migrations	✅
9	Authentication (Basic Auth & OAuth2)	✅
10	Health Checks & Metrics	✅
11	Testing in Dropwizard	✅
12	Lombok – Boilerplate Killer	✅
13	MapStruct – Object Mapping	✅
14	Google Guice – Dependency Injection	✅
15	OkHttp – HTTP Client	✅
16	Hystrix – Circuit Breaker	✅
17	TODO: Advanced Topics	🔲

Chapter 1: What is Dropwizard & Why Use It?

🤔 The Problem It Solves

Imagine you want to build a Java REST API. You'd need to:

Pick a web server (Jetty, Tomcat?)
Pick a JSON library (Jackson, Gson?)
Pick a validation library
Pick a database library
Pick a metrics/monitoring library
Make sure they all work together (version conflicts are a nightmare!)

This research alone can take weeks and paralyze a developer.

✅ Dropwizard's Solution

Dropwizard is a curated collection of best-of-breed libraries glued together so they work out of the box.

Component	Library Used
Web Server	Jetty (embedded)
REST Framework	Jersey (JAX-RS)
JSON	Jackson
Validation	Hibernate Validator
Database Migrations	Liquibase
Metrics & Health	Metrics library
Logging	Logback + SLF4J
Config	YAML + Jackson

🫙 The Fat JAR Concept

Traditional Java apps rely on an app server (Tomcat, GlassFish) being installed on the machine. The problem:

"It works on my computer" 😤

Dropwizard packages everything — your code + all dependencies + the web server — into a single JAR file.

myapp.jar  ← Contains Jetty, Jersey, Jackson, your code... everything!

To run it:

java -jar myapp.jar server config.yml

That's it. No app server needed. No "works on my machine" problem.

🏗️ Microservices Context

Microservices = breaking a big monolithic app into small, independent services. Each service:

Does one thing well
Has its own database
Communicates via REST/HTTP
Can be deployed independently

Dropwizard is perfect for microservices because each service is a single JAR that starts up fast.

🆚 Dropwizard vs Spring Boot

Feature	Dropwizard	Spring Boot
Philosophy	Opinionated, curated	Flexible, extensive
Learning curve	Easier if you know JAX-RS	Easier if you know Spring
Startup speed	Fast	Moderate
Ecosystem size	Smaller	Huge
Best for	REST APIs, microservices	Anything Java

Which to choose? If you're building a focused REST microservice, Dropwizard is excellent. For complex enterprise apps, Spring Boot has more features.

Chapter 2: Project Setup & Fat JAR

🛠️ Prerequisites

Make sure you have installed:

Java 8+ (Dropwizard 1.x requires Java 8)
Maven 3.x
An IDE (IntelliJ IDEA recommended)

📁 Creating a Project with Maven Archetype

The fastest way to start:

mvn archetype:generate \
  -DarchetypeGroupId=io.dropwizard.archetypes \
  -DarchetypeArtifactId=java-simple \
  -DarchetypeVersion=1.3.29 \
  -DgroupId=com.example \
  -DartifactId=my-api \
  -Dversion=1.0-SNAPSHOT \
  -Dname=MyApi

⚠️ Important: The -Dname=MyApi parameter is used in generated file names. Always provide it!

📂 Generated Project Structure

my-api/
├── pom.xml
└── src/
    └── main/
        ├── java/
        │   └── com/example/
        │       ├── MyApiApplication.java   ← Main class (entry point)
        │       └── MyApiConfiguration.java ← Config class
        └── resources/
            └── config.yml                  ← Configuration file

📄 The `pom.xml` — Your Project's Heart

<project>
    <groupId>com.example</groupId>
    <artifactId>my-api</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <dropwizard.version>1.3.29</dropwizard.version>
    </properties>

    <dependencies>
        <!-- Core Dropwizard -->
        <dependency>
            <groupId>io.dropwizard</groupId>
            <artifactId>dropwizard-core</artifactId>
            <version>${dropwizard.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!-- This plugin creates the Fat JAR -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <configuration>
                    <createDependencyReducedPom>true</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals><goal>shade</goal></goals>
                        <configuration>
                            <transformers>
                                <transformer implementation=
                                  "org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                                <transformer implementation=
                                  "org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <!-- This tells Java which class has main() -->
                                    <mainClass>com.example.MyApiApplication</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

🚀 The Application Class

package com.example;

import io.dropwizard.Application;
import io.dropwizard.setup.Bootstrap;
import io.dropwizard.setup.Environment;

public class MyApiApplication extends Application<MyApiConfiguration> {

    // Java's main method — this starts everything
    public static void main(String[] args) throws Exception {
        new MyApiApplication().run(args);
    }

    @Override
    public String getName() {
        return "my-api";  // Name of your app
    }

    @Override
    public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
        // Register bundles (plugins) here
    }

    @Override
    public void run(MyApiConfiguration configuration, Environment environment) {
        // Register your resources (REST endpoints) here
        // This is where the app "comes alive"
    }
}

🏗️ Building & Running

# Build the Fat JAR
mvn clean package

# Run with server command
java -jar target/my-api-1.0-SNAPSHOT.jar server config.yml

# Check if it's alive (admin port 8081)
curl http://localhost:8081/ping

# Your API is on port 8080 by default
curl http://localhost:8080/hello

📋 Built-in Commands

# Validate your config file (catches errors before running!)
java -jar my-api.jar check config.yml

# Run the server
java -jar my-api.jar server config.yml

# Database commands (when Liquibase is added)
java -jar my-api.jar db migrate config.yml
java -jar my-api.jar db status config.yml
java -jar my-api.jar db drop-all --confirm-delete-everything config.yml

💡 Pro tip: Run check before server in deployment scripts to catch config errors early!

Chapter 3: Configuration with YAML

🤷 Why Configuration Matters

Hard-coding things like database URLs, passwords, and ports in code is bad practice. Configuration files let you:

Have different settings for dev/test/production
Change settings without recompiling
Keep secrets out of code

📄 The `config.yml` File

# Server ports
server:
  applicationConnectors:
    - type: http
      port: 8080
  adminConnectors:
    - type: http
      port: 8081

# Logging
logging:
  level: INFO
  loggers:
    com.example: DEBUG  # More verbose logging for your package

# Custom config (your own properties)
login: admin
password: secret123

# Database (added later with Hibernate bundle)
database:
  driverClass: com.mysql.jdbc.Driver
  url: jdbc:mysql://localhost:3306/mydb
  user: dbuser
  password: dbpassword

☕ The Configuration Class

Every key in config.yml maps to a field in your Configuration class:

package com.example;

import io.dropwizard.Configuration;
import com.fasterxml.jackson.annotation.JsonProperty;
import org.hibernate.validator.constraints.NotEmpty;

public class MyApiConfiguration extends Configuration {

    // @NotEmpty = Hibernate Validator annotation
    // If this field is empty/missing, app won't start!
    @NotEmpty
    private String login;

    @NotEmpty
    private String password;

    // @JsonProperty tells Jackson the YAML key name
    @JsonProperty
    public String getLogin() {
        return login;
    }

    @JsonProperty
    public void setLogin(String login) {
        this.login = login;
    }

    @JsonProperty
    public String getPassword() {
        return password;
    }

    @JsonProperty
    public void setPassword(String password) {
        this.password = password;
    }
}

🌍 Multiple Config Files (Best Practice)

# Development
java -jar my-api.jar server config-dev.yml

# Production  
java -jar my-api.jar server config-prod.yml

# Testing (uses H2 in-memory database)
java -jar my-api.jar server config-test.yml

Each config file can point to different databases, use different ports, etc.

🔐 Accessing Config in Your App

@Override
public void run(MyApiConfiguration config, Environment environment) {
    // Access config values
    String login = config.getLogin();
    String password = config.getPassword();

    // Pass them to classes that need them
    environment.jersey().register(new MyResource(login, password));
}

Chapter 4: Building REST Resources (JAX-RS)

🌐 What is JAX-RS?

JAX-RS is a Java standard (specification) for building REST APIs using annotations. Dropwizard uses Jersey as the JAX-RS implementation.

The key idea: annotate a regular Java class with special annotations, and Jersey turns it into a REST endpoint.

📌 Core Annotations

Annotation	Meaning
`@Path("/users")`	URL path this class/method handles
`@GET`	Responds to HTTP GET requests
`@POST`	Responds to HTTP POST requests
`@PUT`	Responds to HTTP PUT requests
`@DELETE`	Responds to HTTP DELETE requests
`@Produces(MediaType.APPLICATION_JSON)`	Returns JSON
`@Consumes(MediaType.APPLICATION_JSON)`	Accepts JSON input
`@PathParam("id")`	Gets value from URL: `/users/42` → id=42
`@QueryParam("name")`	Gets value from query string: `?name=John`

☕ Your First Resource Class

package com.example.resources;

import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.List;
import java.util.ArrayList;

@Path("/bookmarks")         // All methods in this class handle /bookmarks
@Produces(MediaType.APPLICATION_JSON)  // All responses are JSON
@Consumes(MediaType.APPLICATION_JSON)  // All requests accept JSON
public class BookmarkResource {

    private List<Bookmark> bookmarks = new ArrayList<>();

    // GET /bookmarks → returns all bookmarks
    @GET
    public List<Bookmark> getAllBookmarks() {
        return bookmarks;
    }

    // GET /bookmarks/42 → returns one bookmark by ID
    @GET
    @Path("/{id}")
    public Bookmark getBookmark(@PathParam("id") long id) {
        return bookmarks.stream()
            .filter(b -> b.getId() == id)
            .findFirst()
            .orElseThrow(() -> new NotFoundException("Bookmark not found: " + id));
    }

    // POST /bookmarks → create a new bookmark
    @POST
    public Response createBookmark(Bookmark bookmark) {
        bookmarks.add(bookmark);
        return Response.status(Response.Status.CREATED)
                       .entity(bookmark)
                       .build();
    }

    // DELETE /bookmarks/42 → delete a bookmark
    @DELETE
    @Path("/{id}")
    public Response deleteBookmark(@PathParam("id") long id) {
        bookmarks.removeIf(b -> b.getId() == id);
        return Response.noContent().build();  // 204 No Content
    }

    // GET /bookmarks/search?tag=java → filter by query param
    @GET
    @Path("/search")
    public List<Bookmark> searchByTag(@QueryParam("tag") String tag) {
        // filter logic here
        return bookmarks;
    }
}

📦 The Domain Model (Bookmark.java)

package com.example.core;

public class Bookmark {
    private long id;
    private String url;
    private String title;
    private String tag;

    // Default constructor (required for JSON deserialization)
    public Bookmark() {}

    public Bookmark(long id, String url, String title) {
        this.id = id;
        this.url = url;
        this.title = title;
    }

    // Getters and setters
    public long getId() { return id; }
    public void setId(long id) { this.id = id; }
    public String getUrl() { return url; }
    public void setUrl(String url) { this.url = url; }
    public String getTitle() { return title; }
    public void setTitle(String title) { this.title = title; }
    public String getTag() { return tag; }
    public void setTag(String tag) { this.tag = tag; }
}

📝 Register the Resource in Application

@Override
public void run(MyApiConfiguration config, Environment environment) {
    // Without this line, Dropwizard doesn't know about your resource!
    environment.jersey().register(new BookmarkResource());
}

🔧 HTTP Status Codes Quick Reference

Code	Meaning	When to use
200	OK	Successful GET, PUT
201	Created	Successful POST (new resource created)
204	No Content	Successful DELETE
400	Bad Request	Invalid input
401	Unauthorized	Not authenticated
403	Forbidden	Authenticated but no permission
404	Not Found	Resource doesn't exist
500	Server Error	Something blew up

🧑‍💻 Recommended Folder Structure

src/main/java/com/example/
├── MyApiApplication.java       ← Entry point
├── MyApiConfiguration.java     ← Config class
├── core/                       ← Domain models / entities
│   └── Bookmark.java
├── db/                         ← Database access (DAOs)
│   └── BookmarkDAO.java
├── resources/                  ← REST endpoints
│   └── BookmarkResource.java
└── auth/                       ← Authentication
    └── MyAuthenticator.java

Chapter 5: Bean Validation (Hibernate Validator)

❓ Why Validate?

Users send bad data. Always. Never trust input. Validation catches bad data before it hits your database.

Without validation:

{ "url": "", "title": null }

This would save garbage data to your database!

📦 Dependency (already included in dropwizard-core)

Dropwizard includes Hibernate Validator automatically. No extra dependency needed!

🏷️ Common Validation Annotations

import javax.validation.constraints.*;
import org.hibernate.validator.constraints.*;

public class Bookmark {

    private long id;

    @NotEmpty(message = "URL cannot be empty")
    @URL(message = "Must be a valid URL")
    private String url;

    @NotEmpty
    @Size(min = 1, max = 200, message = "Title must be 1-200 characters")
    private String title;

    @NotNull
    private String tag;

    @Min(value = 0, message = "Views cannot be negative")
    private int views;

    @Email(message = "Must be a valid email")
    private String authorEmail;

    @Pattern(regexp = "^[A-Z].*", message = "Must start with uppercase")
    private String category;
}

✅ Using `@Valid` in Resources

Add @Valid before the parameter — Dropwizard validates automatically!

@POST
public Response createBookmark(@Valid Bookmark bookmark) {
    // If bookmark fails validation, Dropwizard returns 422 Unprocessable Entity
    // with details about what's wrong — automatically!
    bookmarks.add(bookmark);
    return Response.status(201).entity(bookmark).build();
}

If validation fails, the client gets:

{
  "errors": [
    "bookmark.url may not be empty",
    "bookmark.title size must be between 1 and 200"
  ]
}

🛡️ Validating Config Too!

public class MyApiConfiguration extends Configuration {

    @NotEmpty  // App won't start if login is missing in config.yml!
    @JsonProperty
    private String login;

    @NotEmpty
    @JsonProperty
    private String password;
}

✍️ Custom Validator

// 1. Create the annotation
@Documented
@Constraint(validatedBy = NoSpacesValidator.class)
@Target({ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
public @interface NoSpaces {
    String message() default "Must not contain spaces";
    Class<?>[] groups() default {};
    Class<? extends Payload>[] payload() default {};
}

// 2. Create the validator logic
public class NoSpacesValidator implements ConstraintValidator<NoSpaces, String> {
    @Override
    public boolean isValid(String value, ConstraintValidatorContext ctx) {
        return value == null || !value.contains(" ");
    }
}

// 3. Use it
public class Bookmark {
    @NoSpaces
    private String slug;
}

Chapter 6: Jackson – JSON Serialization

🔄 What is Jackson?

Jackson converts between Java objects ↔ JSON. This is called serialization (Java → JSON) and deserialization (JSON → Java).

Dropwizard uses Jackson automatically for all REST responses. When your method returns a Bookmark object, Jackson converts it to JSON for you.

📦 Dependency

Already included in dropwizard-core!

🏷️ Key Jackson Annotations

import com.fasterxml.jackson.annotation.*;

public class Bookmark {

    // Changes the JSON key name
    @JsonProperty("bookmark_id")
    private long id;  // Java field "id" → JSON key "bookmark_id"

    // Include in JSON even if null
    @JsonInclude(JsonInclude.Include.ALWAYS)
    private String title;

    // Exclude from JSON output
    @JsonIgnore
    private String internalNotes;

    // Exclude null values from JSON
    @JsonInclude(JsonInclude.Include.NON_NULL)
    private String optionalField;

    // Custom date format
    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss")
    private Date createdAt;

    // Map JSON → object (deserialization constructor)
    @JsonCreator
    public Bookmark(@JsonProperty("bookmark_id") long id,
                    @JsonProperty("url") String url) {
        this.id = id;
        this.url = url;
    }
}

🗺️ Working with Unknown Fields

// Ignore any extra JSON fields you don't have a field for
@JsonIgnoreProperties(ignoreUnknown = true)
public class Bookmark {
    private String url;
    // JSON might have "extra_field" — this annotation ignores it instead of throwing an error
}

⚙️ Customizing Jackson in Dropwizard

@Override
public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
    // Customize the ObjectMapper
    bootstrap.getObjectMapper()
        .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
        .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
        .setSerializationInclusion(JsonInclude.Include.NON_NULL);
}

🌐 Polymorphic Types (Advanced)

// Base class
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes({
    @JsonSubTypes.Type(value = WebBookmark.class, name = "web"),
    @JsonSubTypes.Type(value = BookBookmark.class, name = "book")
})
public abstract class Bookmark { }

public class WebBookmark extends Bookmark { private String url; }
public class BookBookmark extends Bookmark { private String isbn; }

// JSON will look like:
// { "type": "web", "url": "https://..." }
// { "type": "book", "isbn": "978-..." }

Chapter 7: Database Access with Hibernate/JDBI

🗃️ Your Two Options in Dropwizard

Option	Description	Best For
Hibernate (JPA)	ORM — maps Java objects to DB tables	Complex domain models
JDBI	Thin wrapper over JDBC — write SQL directly	Simple, fast queries

📦 Dependencies

<!-- For Hibernate -->
<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-hibernate</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

<!-- MySQL Driver (use 5.x to avoid Liquibase conflicts!) -->
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>5.1.49</version>
</dependency>

🗄️ Part A: Hibernate

Step 1 — The Entity (maps to a DB table)

package com.example.core;

import javax.persistence.*;

@Entity                         // This Java class = a database table
@Table(name = "bookmarks")      // Table name in the database
@NamedQueries({
    // Pre-defined queries attached to this class
    @NamedQuery(
        name = "com.example.core.Bookmark.findAll",
        query = "SELECT b FROM Bookmark b"     // JPQL (not SQL!)
    ),
    @NamedQuery(
        name = "com.example.core.Bookmark.findByTag",
        query = "SELECT b FROM Bookmark b WHERE b.tag = :tag"
    )
})
public class Bookmark {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)  // Auto-increment
    private long id;

    @Column(name = "url", nullable = false)
    private String url;

    @Column(name = "title")
    private String title;

    @Column(name = "tag")
    private String tag;

    // Getters, setters, default constructor...
}

Step 2 — The DAO (Data Access Object)

package com.example.db;

import io.dropwizard.hibernate.AbstractDAO;
import com.example.core.Bookmark;
import org.hibernate.SessionFactory;
import org.hibernate.query.Query;
import java.util.List;
import java.util.Optional;

public class BookmarkDAO extends AbstractDAO<Bookmark> {

    public BookmarkDAO(SessionFactory factory) {
        super(factory);  // AbstractDAO handles Hibernate session management
    }

    // Get all bookmarks
    public List<Bookmark> findAll() {
        return list(namedQuery("com.example.core.Bookmark.findAll"));
    }

    // Get by ID — returns Optional (empty if not found)
    public Optional<Bookmark> findById(long id) {
        return Optional.ofNullable(get(id));
    }

    // Get by tag — using named query parameter
    public List<Bookmark> findByTag(String tag) {
        Query<Bookmark> query = namedQuery("com.example.core.Bookmark.findByTag");
        query.setParameter("tag", tag);
        return list(query);
    }

    // Create or update
    public Bookmark save(Bookmark bookmark) {
        return persist(bookmark);  // AbstractDAO's persist = INSERT or UPDATE
    }

    // Delete
    public void delete(Bookmark bookmark) {
        currentSession().delete(bookmark);
    }
}

Step 3 — Register Hibernate Bundle in Application

public class MyApiApplication extends Application<MyApiConfiguration> {

    // Create the Hibernate bundle — tell it about all your entities!
    private final HibernateBundle<MyApiConfiguration> hibernate =
        new HibernateBundle<MyApiConfiguration>(Bookmark.class) {  // list ALL entities here
            @Override
            public DataSourceFactory getDataSourceFactory(MyApiConfiguration config) {
                return config.getDataSourceFactory();
            }
        };

    @Override
    public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
        bootstrap.addBundle(hibernate);  // Register the bundle!
    }

    @Override
    public void run(MyApiConfiguration config, Environment environment) {
        // Create DAO with Hibernate session factory
        final BookmarkDAO dao = new BookmarkDAO(hibernate.getSessionFactory());

        // Pass DAO to resource
        environment.jersey().register(new BookmarkResource(dao));
    }
}

Step 4 — Config file addition

# config.yml
database:
  driverClass: com.mysql.jdbc.Driver
  url: jdbc:mysql://localhost:3306/bookmarks_db
  user: root
  password: secret
  properties:
    charSet: UTF-8
    hibernate.dialect: org.hibernate.dialect.MySQL5InnoDBDialect
  maxWaitForConnection: 1s
  validationQuery: "/* MyService Health Check */ SELECT 1"
  minSize: 2
  maxSize: 8
  checkConnectionWhileIdle: false

Step 5 — Add DataSourceFactory to Config class

public class MyApiConfiguration extends Configuration {

    @Valid
    @NotNull
    private DataSourceFactory database = new DataSourceFactory();

    @JsonProperty("database")
    public DataSourceFactory getDataSourceFactory() { return database; }

    @JsonProperty("database")
    public void setDataSourceFactory(DataSourceFactory factory) {
        this.database = factory;
    }
}

Step 6 — Use `@UnitOfWork` in Resource

public class BookmarkResource {
    private final BookmarkDAO dao;

    public BookmarkResource(BookmarkDAO dao) {
        this.dao = dao;
    }

    @GET
    // @UnitOfWork manages the Hibernate session + transaction for you!
    // Without it, you'd have to open/close sessions manually.
    @UnitOfWork
    public List<Bookmark> getAllBookmarks() {
        return dao.findAll();
        // ⚠️ WARNING: Lazy-loaded fields must be accessed INSIDE this method!
    }

    @POST
    @UnitOfWork
    public Bookmark createBookmark(@Valid Bookmark bookmark) {
        return dao.save(bookmark);
    }
}

Chapter 8: Liquibase – Database Migrations

🤔 The Problem Without Migrations

Imagine two developers both changing the database schema. Developer A adds a tag column, Developer B renames url to link. Who goes first? How do you track DB changes? How do you roll back?

Liquibase is like Git, but for your database schema.

📦 Dependency

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-migrations</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

⚙️ Register the Bundle

@Override
public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
    bootstrap.addBundle(new MigrationsBundle<MyApiConfiguration>() {
        @Override
        public DataSourceFactory getDataSourceFactory(MyApiConfiguration config) {
            return config.getDataSourceFactory();
        }
    });
}

📄 The Migration File (`migrations.xml`)

Place at src/main/resources/migrations.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">

    <!-- Each changeSet = one atomic database change -->
    <!-- id + author combination must be globally unique! -->

    <changeSet id="1" author="mitri">
        <createTable tableName="bookmarks">
            <column name="id" type="BIGINT" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="url" type="VARCHAR(500)">
                <constraints nullable="false"/>
            </column>
            <column name="title" type="VARCHAR(255)"/>
            <column name="tag" type="VARCHAR(100)"/>
        </createTable>
        <!-- Liquibase can auto-generate rollback for createTable (just DROP TABLE) -->
    </changeSet>

    <changeSet id="2" author="mitri">
        <addColumn tableName="bookmarks">
            <column name="created_at" type="TIMESTAMP" defaultValueDate="CURRENT_TIMESTAMP"/>
        </addColumn>
    </changeSet>

    <changeSet id="3" author="mitri">
        <!-- Insert test data — must specify manual rollback! -->
        <insert tableName="bookmarks">
            <column name="url" value="https://dropwizard.io"/>
            <column name="title" value="Dropwizard Official Site"/>
            <column name="tag" value="java"/>
        </insert>
        <rollback>
            <!-- Tell Liquibase how to undo this changeSet -->
            <delete tableName="bookmarks">
                <where>url = 'https://dropwizard.io'</where>
            </delete>
        </rollback>
    </changeSet>

</databaseChangeLog>

🚀 Running Migrations

# Check current state of your database
java -jar my-api.jar db status config.yml

# Apply all pending migrations
java -jar my-api.jar db migrate config.yml

# Apply migrations up to a specific tag
java -jar my-api.jar db migrate --include-tag mytag config.yml

# Roll back last 1 change
java -jar my-api.jar db rollback --count 1 config.yml

# Danger zone! Drops everything. Only for dev!
java -jar my-api.jar db drop-all --confirm-delete-everything config.yml

🆚 Liquibase vs Flyway

Feature	Liquibase	Flyway
Migration format	XML, YAML, SQL, JSON	SQL (primarily)
Rollbacks	Built-in support	Manual in Flyway Pro
Author tracking	Yes (prevents conflicts)	No
DB independence	Yes (XML format)	SQL-only = DB specific
Dropwizard default	✅ Yes	Via third-party bundle

Why Liquibase wins for teams: The author field prevents naming collisions. Two developers can't accidentally create conflicting migrations.

Chapter 9: Authentication

🔐 Dropwizard's Two Built-in Auth Methods

Basic Auth — Username + Password sent with every request
OAuth 2.0 — Token-based auth (used by most modern APIs)

📦 Dependency

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-auth</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

👤 Step 1 — The User Class

Must extend java.security.Principal in Dropwizard 1.x:

package com.example.auth;

import java.security.Principal;

public class User implements Principal {
    private final String name;

    public User(String name) {
        this.name = name;
    }

    @Override
    public String getName() {
        return name;
    }
}

🔑 Step 2 — The Authenticator

package com.example.auth;

import io.dropwizard.auth.AuthenticationException;
import io.dropwizard.auth.Authenticator;
import io.dropwizard.auth.basic.BasicCredentials;
import java.util.Optional;

public class MyAuthenticator implements Authenticator<BasicCredentials, User> {

    private final String validLogin;
    private final String validPassword;

    public MyAuthenticator(String login, String password) {
        this.validLogin = login;
        this.validPassword = password;
    }

    @Override
    public Optional<User> authenticate(BasicCredentials credentials)
            throws AuthenticationException {

        // Check if credentials match
        if (validLogin.equals(credentials.getUsername()) &&
            validPassword.equals(credentials.getPassword())) {
            return Optional.of(new User(credentials.getUsername()));
        }

        // Return empty Optional = authentication failed
        // Dropwizard will return 401 Unauthorized automatically
        return Optional.empty();
    }
}

🏭 Step 3 — Register in Application

@Override
public void run(MyApiConfiguration config, Environment environment) {

    // Create the authenticator with credentials from config
    MyAuthenticator authenticator = new MyAuthenticator(
        config.getLogin(),
        config.getPassword()
    );

    // Register Basic Auth with the authenticator
    environment.jersey().register(
        new AuthDynamicFeature(
            new BasicCredentialAuthFilter.Builder<User>()
                .setAuthenticator(authenticator)
                .setRealm("My Secure API")  // Shown in browser login dialog
                .buildAuthFilter()
        )
    );

    // Enable @Auth annotation injection in resources
    environment.jersey().register(new AuthValueFactoryProvider.Binder<>(User.class));
}

🔒 Step 4 — Protect Your Endpoints

@GET
@Path("/{id}")
@UnitOfWork
// @Auth injects the authenticated user AND protects the endpoint
public Bookmark getBookmark(@Auth User user,  // user = logged in user
                             @PathParam("id") long id) {
    // Only authenticated users reach here
    return dao.findById(id)
              .orElseThrow(() -> new NotFoundException("Not found: " + id));
}

📡 Making a Request with Basic Auth

# Using curl — -u "username:password"
curl -u "admin:secret123" http://localhost:8080/bookmarks/1

# With HTTPS (ignore self-signed cert for dev)
curl -k -u "admin:secret123" https://localhost:8443/bookmarks

# Or manually with header
curl -H "Authorization: Basic YWRtaW46c2VjcmV0MTIz" http://localhost:8080/bookmarks

Chapter 10: Health Checks & Metrics

💓 Health Checks

Health checks answer: "Is my app healthy right now?"

The admin port (8081) exposes:

/ping → returns "pong" if app is up
/healthcheck → runs all registered health checks

Built-in Health Checks

Dropwizard provides one by default:

DeadlockHealthCheck — checks if JVM has thread deadlocks

When you add the Hibernate bundle, you get a free extra:

Database connectivity check — tries to ping the database

Writing a Custom Health Check

package com.example.health;

import com.codahale.metrics.health.HealthCheck;

public class ExternalServiceHealthCheck extends HealthCheck {

    private final String serviceUrl;

    public ExternalServiceHealthCheck(String serviceUrl) {
        this.serviceUrl = serviceUrl;
    }

    @Override
    protected Result check() throws Exception {
        // Try to call the external service
        try {
            // ... make HTTP call to serviceUrl ...
            boolean isReachable = pingService(serviceUrl);

            if (isReachable) {
                return Result.healthy("External service is UP");
            } else {
                return Result.unhealthy("External service is DOWN at: " + serviceUrl);
            }
        } catch (Exception e) {
            return Result.unhealthy("Exception connecting: " + e.getMessage());
        }
    }

    private boolean pingService(String url) {
        // Simplified — use OkHttp (Chapter 15) in real code
        return true;
    }
}

Registering a Health Check

@Override
public void run(MyApiConfiguration config, Environment environment) {
    // Register your custom health check
    environment.healthChecks().register(
        "external-service",
        new ExternalServiceHealthCheck("https://api.thirdparty.com/ping")
    );
}

Response When Healthy

curl http://localhost:8081/healthcheck

{
  "deadlocks": { "healthy": true },
  "hibernate": { "healthy": true },
  "external-service": { "healthy": true, "message": "External service is UP" }
}

Response When Unhealthy (returns 500)

{
  "external-service": {
    "healthy": false,
    "message": "External service is DOWN at: https://api.thirdparty.com/ping"
  }
}

📊 Metrics

Dropwizard's Metrics library lets you measure things in your app:

import com.codahale.metrics.*;

public class BookmarkResource {
    private final Meter requests;      // How many requests per second
    private final Timer latency;       // How long requests take
    private final Counter activeUsers; // Count something
    private final Histogram payloadSize; // Distribution of values

    public BookmarkResource(MetricRegistry metrics, BookmarkDAO dao) {
        this.requests = metrics.meter("bookmark.requests");
        this.latency = metrics.timer("bookmark.latency");
        this.activeUsers = metrics.counter("bookmark.active-users");
    }

    @GET
    public List<Bookmark> getAllBookmarks() {
        requests.mark();  // Record a request happened
        try (Timer.Context ctx = latency.time()) {  // Measure how long this takes
            return dao.findAll();
        }
    }
}

📈 Sending Metrics to Graphite

# config.yml
metrics:
  reporters:
    - type: graphite
      host: graphite.example.com
      port: 2003
      prefix: myapp
      frequency: 1 minute

Chapter 11: Testing in Dropwizard

🧪 Testing Philosophy

Dropwizard takes testing seriously and provides special tools that make it easy to test REST APIs without running a full server.

📦 Dependency

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-testing</artifactId>
    <version>${dropwizard.version}</version>
    <scope>test</scope>
</dependency>
<!-- Note: JUnit is included automatically! No need to add it separately. -->

🧩 Level 1: Unit Testing a Resource (ResourceTestRule)

Tests just your resource class + Jersey, without a real database:

package com.example.resources;

import io.dropwizard.testing.junit.ResourceTestRule;
import org.junit.*;
import static org.mockito.Mockito.*;
import static org.assertj.core.api.Assertions.*;
import javax.ws.rs.core.Response;
import java.util.List;

public class BookmarkResourceTest {

    // Mock the DAO — no real database needed!
    private static final BookmarkDAO dao = mock(BookmarkDAO.class);

    // Spins up an in-memory Jersey server for testing
    @ClassRule
    public static final ResourceTestRule resources = ResourceTestRule.builder()
            .addResource(new BookmarkResource(dao))
            .build();

    @Before
    public void setup() {
        // Define what the mock returns
        when(dao.findAll()).thenReturn(List.of(new Bookmark(1L, "https://example.com", "Example")));
    }

    @After
    public void tearDown() {
        reset(dao);  // Reset mock between tests
    }

    @Test
    public void testGetAllBookmarks() {
        // Make a GET request using the test client
        List<Bookmark> result = resources.client()
                .target("/bookmarks")
                .request()
                .get(new GenericType<List<Bookmark>>(){});

        assertThat(result).hasSize(1);
        assertThat(result.get(0).getUrl()).isEqualTo("https://example.com");
        verify(dao).findAll();  // Verify DAO was called
    }

    @Test
    public void testCreateBookmark_returns201() {
        Bookmark newBookmark = new Bookmark(0, "https://new.com", "New Site");
        when(dao.save(any(Bookmark.class))).thenReturn(newBookmark);

        Response response = resources.client()
                .target("/bookmarks")
                .request()
                .post(Entity.json(newBookmark));

        assertThat(response.getStatus()).isEqualTo(201);
    }

    @Test
    public void testGetBookmark_notFound_returns404() {
        when(dao.findById(99L)).thenReturn(Optional.empty());

        Response response = resources.client()
                .target("/bookmarks/99")
                .request()
                .get();

        assertThat(response.getStatus()).isEqualTo(404);
    }
}

🔗 Level 2: Integration Testing (DropwizardAppRule)

Starts the entire application — tests everything together:

package com.example;

import io.dropwizard.testing.junit.DropwizardAppRule;
import org.junit.*;
import javax.ws.rs.client.*;
import javax.ws.rs.core.Response;

public class BookmarkIntegrationTest {

    // Starts the ENTIRE app (real server, real database)
    @ClassRule
    public static final DropwizardAppRule<MyApiConfiguration> RULE =
        new DropwizardAppRule<>(
            MyApiApplication.class,
            "config-test.yml"  // Use test config (H2 in-memory DB)
        );

    private Client client;

    @Before
    public void setup() {
        client = ClientBuilder.newClient();
    }

    @Test
    public void testServerIsRunning() {
        Response response = client
            .target(String.format("http://localhost:%d/bookmarks",
                RULE.getLocalPort()))
            .request()
            .get();

        assertThat(response.getStatus()).isEqualTo(200);
    }

    @Test
    public void testAuthenticatedEndpoint() {
        Response response = client
            .target(String.format("https://localhost:%d/bookmarks",
                RULE.getLocalPort()))
            .request()
            // Add Basic Auth header
            .header("Authorization", "Basic " + Base64.encode("admin:secret"))
            .get();

        assertThat(response.getStatus()).isEqualTo(200);
    }
}

🗃️ Testing Database Queries (DAOTest)

public class BookmarkDAOTest {

    // Spins up real Hibernate with H2 in-memory database
    @ClassRule
    public static final DAOTestRule database = DAOTestRule.newBuilder()
        .addEntityClass(Bookmark.class)
        .build();

    private BookmarkDAO dao;

    @Before
    public void setUp() throws Exception {
        dao = new BookmarkDAO(database.getSessionFactory());
    }

    @Test
    public void testSaveAndFind() {
        Bookmark saved = database.inTransaction(() -> {
            Bookmark b = new Bookmark();
            b.setUrl("https://test.com");
            b.setTitle("Test");
            return dao.save(b);
        });

        assertThat(saved.getId()).isGreaterThan(0);
        assertThat(dao.findById(saved.getId())).isPresent();
    }
}

Chapter 12: Lombok – Boilerplate Killer

😤 The Problem

Java is verbose. For every field in a class, you write:

A getter
A setter
Constructor(s)
equals() and hashCode()
toString()

That's 30+ lines of boring code for a simple 5-field class.

✨ Lombok's Solution

Annotations that generate all that code at compile time. You write the fields. Lombok writes the rest.

📦 Dependency

<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.30</version>
    <scope>provided</scope>  <!-- Only needed at compile time -->
</dependency>

Also install the Lombok plugin in IntelliJ IDEA (Settings → Plugins → search "Lombok").

🏷️ Core Lombok Annotations

import lombok.*;
import lombok.extern.slf4j.Slf4j;

// @Data = @Getter + @Setter + @ToString + @EqualsAndHashCode + @RequiredArgsConstructor
@Data
public class Bookmark {
    private long id;
    private String url;
    private String title;
}
// Lombok generates: getters, setters, toString(), equals(), hashCode(), constructor

Breakdown of each annotation:

@Getter              // Generates getters for all fields
@Setter              // Generates setters for all fields
@ToString            // Generates toString() like: Bookmark(id=1, url=https://...)
@EqualsAndHashCode   // Generates equals() and hashCode() based on fields
@NoArgsConstructor   // Generates: public Bookmark() {}
@AllArgsConstructor  // Generates: public Bookmark(long id, String url, String title) {}
@RequiredArgsConstructor  // Constructor for @NonNull and final fields only
public class Bookmark {
    private long id;
    private String url;
    private String title;
}

🏗️ @builder Pattern

Immutable objects with a fluent builder API:

@Builder
@Getter  // @Builder doesn't auto-generate getters
public class BookmarkRequest {
    private final String url;
    private final String title;
    private final String tag;
}

// Usage:
BookmarkRequest request = BookmarkRequest.builder()
    .url("https://example.com")
    .title("Example")
    .tag("java")
    .build();

🔒 Immutable Objects with @Value

// @Value = immutable version of @Data
// All fields are private final, only getters generated (no setters!)
@Value
public class UserId {
    long id;
    String username;
}

📝 Logging with @Slf4j

@Slf4j  // Generates: private static final Logger log = LoggerFactory.getLogger(BookmarkResource.class);
public class BookmarkResource {

    @GET
    public List<Bookmark> getAll() {
        log.info("Fetching all bookmarks");  // Use log directly!
        log.debug("Debug details here");
        log.error("Something went wrong", exception);
        return dao.findAll();
    }
}

⚠️ Lombok + Jackson Integration

Jackson needs a default constructor for deserialization. Use both:

@Data
@NoArgsConstructor          // Jackson needs this
@AllArgsConstructor         // Useful for your code
@JsonIgnoreProperties(ignoreUnknown = true)
public class Bookmark {
    private long id;
    private String url;
    private String title;
}

Or use @Builder with @Jacksonized:

@Builder
@Jacksonized  // Tells Jackson to use the builder for deserialization
@Value
public class Bookmark {
    long id;
    String url;
    String title;
}

Chapter 13: MapStruct – Object Mapping

🗺️ The Mapping Problem

In real applications, you have different object types:

Entity — maps to database (has Hibernate annotations)
DTO (Data Transfer Object) — what you send/receive in API
Domain Model — internal business logic object

You need to convert between them constantly. Manually writing conversion code is error-prone and tedious.

// Manual mapping — boring, error-prone
public BookmarkDTO toDTO(Bookmark entity) {
    BookmarkDTO dto = new BookmarkDTO();
    dto.setId(entity.getId());
    dto.setUrl(entity.getUrl());
    dto.setTitle(entity.getTitle());
    // Forget one field and you have a bug!
    return dto;
}

✨ MapStruct Solution

MapStruct generates these conversion methods at compile time using annotations.

📦 Dependency

<dependency>
    <groupId>org.mapstruct</groupId>
    <artifactId>mapstruct</artifactId>
    <version>1.5.5.Final</version>
</dependency>
<dependency>
    <groupId>org.mapstruct</groupId>
    <artifactId>mapstruct-processor</artifactId>
    <version>1.5.5.Final</version>
    <scope>provided</scope>
</dependency>

🔧 Maven Plugin (required with Lombok)

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <annotationProcessorPaths>
            <path>
                <groupId>org.projectlombok</groupId>
                <artifactId>lombok</artifactId>
                <version>1.18.30</version>
            </path>
            <path>
                <groupId>org.mapstruct</groupId>
                <artifactId>mapstruct-processor</artifactId>
                <version>1.5.5.Final</version>
            </path>
        </annotationProcessorPaths>
    </configuration>
</plugin>

📦 Define Your Objects

// Database Entity
@Entity
@Data
public class BookmarkEntity {
    @Id @GeneratedValue
    private Long id;
    private String url;
    private String title;
    private String tag;
    private Date createdAt;
    private String internalNotes;  // Should NOT be in API response!
}

// API DTO (Data Transfer Object)
@Data
@NoArgsConstructor
public class BookmarkDTO {
    private Long id;
    private String url;
    private String title;
    private String tag;
    // Notice: no internalNotes — we don't expose this!
    private String createdDate;  // Different format than entity!
}

🗺️ The Mapper Interface

package com.example.mapper;

import org.mapstruct.*;

@Mapper(componentModel = "default")  // Use "cdi" for CDI, "spring" for Spring
public interface BookmarkMapper {

    // Simple mapping — same field names map automatically!
    BookmarkDTO toDTO(BookmarkEntity entity);

    // Map from DTO back to entity
    BookmarkEntity toEntity(BookmarkDTO dto);

    // Custom field mapping
    @Mapping(source = "createdAt", target = "createdDate",
             dateFormat = "yyyy-MM-dd")  // Convert Date to formatted String
    @Mapping(target = "internalNotes", ignore = true)  // Don't map this field
    BookmarkDTO toDTOCustom(BookmarkEntity entity);

    // Map a list automatically!
    List<BookmarkDTO> toDTOList(List<BookmarkEntity> entities);
}

🏭 Getting the Mapper Instance

// Approach 1: Factory method (no DI)
BookmarkMapper mapper = Mappers.getMapper(BookmarkMapper.class);

// Approach 2: With Guice (Chapter 14)
// Bind in your module, inject where needed

✅ Using the Mapper in Resource

@Path("/bookmarks")
public class BookmarkResource {
    private final BookmarkDAO dao;
    private final BookmarkMapper mapper = Mappers.getMapper(BookmarkMapper.class);

    @GET
    @UnitOfWork
    public List<BookmarkDTO> getAllBookmarks() {
        List<BookmarkEntity> entities = dao.findAll();
        return mapper.toDTOList(entities);  // Entity → DTO, one line!
    }

    @POST
    @UnitOfWork
    public BookmarkDTO createBookmark(@Valid BookmarkDTO dto) {
        BookmarkEntity entity = mapper.toEntity(dto);  // DTO → Entity
        BookmarkEntity saved = dao.save(entity);
        return mapper.toDTO(saved);  // Entity → DTO for response
    }
}

Chapter 14: Google Guice – Dependency Injection

🤔 The Problem

As your app grows, you end up with this mess in your run() method:

public void run(Config config, Environment env) {
    HibernateBundle hib = hibernate;
    BookmarkDAO dao = new BookmarkDAO(hib.getSessionFactory());
    BookmarkMapper mapper = Mappers.getMapper(BookmarkMapper.class);
    MyAuthenticator auth = new MyAuthenticator(config.getLogin(), config.getPassword());
    BookmarkResource resource = new BookmarkResource(dao, mapper, auth);
    env.jersey().register(resource);
    // What if BookmarkResource needs 5 more dependencies?!
}

Every class manually creates its dependencies. Change one thing → change everywhere.

✨ Guice Solution

Inversion of Control: Instead of classes creating their dependencies, Guice creates them and injects them where needed.

📦 Dependency

<dependency>
    <groupId>com.google.inject</groupId>
    <artifactId>guice</artifactId>
    <version>5.1.0</version>
</dependency>
<!-- Dropwizard + Guice integration -->
<dependency>
    <groupId>ru.vyarus</groupId>
    <artifactId>dropwizard-guicey</artifactId>
    <version>5.7.1</version>
</dependency>

📝 Step 1 — Mark Dependencies with @Inject

// BookmarkDAO depends on SessionFactory
public class BookmarkDAO extends AbstractDAO<BookmarkEntity> {

    @Inject  // Guice will provide SessionFactory
    public BookmarkDAO(SessionFactory factory) {
        super(factory);
    }
}

// BookmarkResource depends on DAO and Mapper
public class BookmarkResource {

    private final BookmarkDAO dao;
    private final BookmarkMapper mapper;

    @Inject  // Guice will inject both!
    public BookmarkResource(BookmarkDAO dao, BookmarkMapper mapper) {
        this.dao = dao;
        this.mapper = mapper;
    }
}

📋 Step 2 — The Guice Module (Wiring Configuration)

package com.example;

import com.google.inject.AbstractModule;
import com.google.inject.Provides;
import com.google.inject.Singleton;

public class MyModule extends AbstractModule {

    @Override
    protected void configure() {
        // Bind interface → implementation
        // "Whenever someone asks for UserService, give them UserServiceImpl"
        bind(UserService.class).to(UserServiceImpl.class).in(Singleton.class);

        // Bind a class directly
        bind(BookmarkDAO.class).in(Singleton.class);

        // Bind mapper
        bind(BookmarkMapper.class).toInstance(Mappers.getMapper(BookmarkMapper.class));
    }

    // @Provides methods create instances that need configuration
    @Provides
    @Singleton
    public BookmarkDAO provideBookmarkDAO(SessionFactory factory) {
        return new BookmarkDAO(factory);
    }
}

🏭 Step 3 — Register with Dropwizard

Using dropwizard-guicey:

public class MyApiApplication extends GuiceApplication<MyApiConfiguration> {

    @Override
    public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
        super.initialize(bootstrap);
        bootstrap.addBundle(hibernate);
    }

    @Override
    public void run(MyApiConfiguration config, Environment environment) {
        // Resources are auto-discovered and injected!
    }

    @Override
    protected Injector createInjector(MyApiConfiguration config) {
        return Guice.createInjector(new MyModule());
    }
}

🏷️ Key Guice Annotations

@Inject     // Mark constructor/field/method for injection
@Singleton  // One instance for entire app lifetime
@Named("dbUrl")  // Disambiguate when multiple bindings of same type exist

// Usage:
@Inject @Named("dbUrl") String databaseUrl;

🔄 Guice Scopes

bind(BookmarkDAO.class).in(Singleton.class);    // One instance per app
bind(BookmarkDAO.class).in(RequestScoped.class); // New instance per HTTP request
bind(BookmarkDAO.class);                         // Default: new instance every time

Chapter 15: OkHttp – HTTP Client

🌐 Why OkHttp?

Microservices talk to each other over HTTP. You need an HTTP client to:

Call third-party APIs (payment, email, maps)
Call other microservices in your system

OkHttp is the most popular Java HTTP client — fast, simple, feature-rich.

📦 Dependency

<dependency>
    <groupId>com.squareup.okhttp3</groupId>
    <artifactId>okhttp</artifactId>
    <version>4.11.0</version>
</dependency>

🚀 Basic GET Request

import okhttp3.*;
import java.io.IOException;

public class ExternalApiClient {

    // IMPORTANT: Reuse the same OkHttpClient — it manages connection pooling!
    private final OkHttpClient client = new OkHttpClient();
    private final ObjectMapper objectMapper = new ObjectMapper();

    public String fetchData(String url) throws IOException {
        // Build the request
        Request request = new Request.Builder()
            .url(url)
            .header("Accept", "application/json")
            .header("Authorization", "Bearer my-token")
            .build();

        // Execute synchronously
        try (Response response = client.newCall(request).execute()) {
            if (!response.isSuccessful()) {
                throw new IOException("Unexpected response: " + response);
            }
            return response.body().string();
        }
    }
}

📤 POST Request with JSON Body

public static final MediaType JSON = MediaType.get("application/json; charset=utf-8");

public Bookmark createRemoteBookmark(BookmarkDTO dto) throws IOException {
    String json = objectMapper.writeValueAsString(dto);

    Request request = new Request.Builder()
        .url("https://api.other-service.com/bookmarks")
        .post(RequestBody.create(json, JSON))  // POST with JSON body
        .build();

    try (Response response = client.newCall(request).execute()) {
        String responseBody = response.body().string();
        return objectMapper.readValue(responseBody, Bookmark.class);
    }
}

⏱️ Timeouts & Configuration

OkHttpClient client = new OkHttpClient.Builder()
    .connectTimeout(10, TimeUnit.SECONDS)  // How long to wait to connect
    .readTimeout(30, TimeUnit.SECONDS)     // How long to wait for data
    .writeTimeout(15, TimeUnit.SECONDS)    // How long to wait for upload
    .retryOnConnectionFailure(true)        // Retry on network errors
    .build();

🔄 Async Requests

public void fetchDataAsync(String url, Callback callback) {
    Request request = new Request.Builder().url(url).build();

    // Non-blocking — callback is called when done
    client.newCall(request).enqueue(new Callback() {
        @Override
        public void onFailure(Call call, IOException e) {
            log.error("Request failed", e);
        }

        @Override
        public void onResponse(Call call, Response response) throws IOException {
            try (response) {
                String body = response.body().string();
                // Process response
            }
        }
    });
}

🔌 OkHttp Interceptors (Middleware)

Interceptors run for every request — perfect for logging, auth headers, etc.:

// Logging interceptor
OkHttpClient client = new OkHttpClient.Builder()
    .addInterceptor(chain -> {
        Request request = chain.request();
        log.info("→ {} {}", request.method(), request.url());

        long start = System.currentTimeMillis();
        Response response = chain.proceed(request);
        long duration = System.currentTimeMillis() - start;

        log.info("← {} {} ({}ms)", response.code(), request.url(), duration);
        return response;
    })
    .addInterceptor(chain -> {
        // Auth interceptor — add token to every request
        Request authenticated = chain.request().newBuilder()
            .header("Authorization", "Bearer " + getToken())
            .build();
        return chain.proceed(authenticated);
    })
    .build();

Chapter 16: Hystrix – Circuit Breaker

⚡ The Cascading Failure Problem

Service A calls Service B. Service B calls Service C. Service C goes down.

Without circuit breakers:

Service C times out after 30 seconds
Service B waits 30s, then times out
Service A waits 30s too = everything hangs for 30+ seconds!
Threads pile up, memory fills up, your whole system crashes

🔌 The Circuit Breaker Pattern

Like an electrical circuit breaker — when failures exceed a threshold, it "trips" and subsequent calls fail immediately instead of waiting.

States:

CLOSED — Normal operation, calls pass through
OPEN — Too many failures, all calls fail immediately
HALF-OPEN — Testing if service recovered; allows one call through

📦 Dependency

<dependency>
    <groupId>com.netflix.hystrix</groupId>
    <artifactId>hystrix-core</artifactId>
    <version>1.5.18</version>
</dependency>

☕ Wrapping Calls with HystrixCommand

package com.example.hystrix;

import com.netflix.hystrix.*;
import java.io.IOException;

public class FetchBookmarkCommand extends HystrixCommand<String> {

    private final String bookmarkId;
    private final ExternalApiClient apiClient;

    public FetchBookmarkCommand(String bookmarkId, ExternalApiClient apiClient) {
        super(HystrixCommandGroupKey.Factory.asKey("BookmarkService"));
        this.bookmarkId = bookmarkId;
        this.apiClient = apiClient;
    }

    @Override
    protected String run() throws Exception {
        // The actual (potentially failing) call
        return apiClient.fetchData("https://api.service.com/bookmarks/" + bookmarkId);
    }

    @Override
    protected String getFallback() {
        // This runs when:
        // 1. run() throws an exception
        // 2. Circuit is OPEN (previous failures)
        // 3. Timeout occurs
        return "{\"id\":\"" + bookmarkId + "\",\"status\":\"unavailable\"}";
    }
}

🎯 Using the Command

// Synchronous execution
String result = new FetchBookmarkCommand(bookmarkId, client).execute();

// Asynchronous (returns Future)
Future<String> future = new FetchBookmarkCommand(bookmarkId, client).queue();

// Reactive (returns Observable)
Observable<String> observable = new FetchBookmarkCommand(bookmarkId, client).observe();

⚙️ Configuring Hystrix

HystrixCommand.Setter config = HystrixCommand.Setter
    .withGroupKey(HystrixCommandGroupKey.Factory.asKey("BookmarkService"))
    .andCommandKey(HystrixCommandKey.Factory.asKey("FetchBookmark"))
    .andCommandPropertiesDefaults(
        HystrixCommandProperties.Setter()
            .withExecutionTimeoutInMilliseconds(3000)  // 3 second timeout
            .withCircuitBreakerRequestVolumeThreshold(10)  // Min requests before circuit can trip
            .withCircuitBreakerErrorThresholdPercentage(50)  // Trip if 50% fail
            .withCircuitBreakerSleepWindowInMilliseconds(5000)  // Wait 5s before trying again
    );

// Use in command:
public FetchBookmarkCommand(String id, ExternalApiClient client) {
    super(config);  // Pass config to super
    this.bookmarkId = id;
    this.apiClient = client;
}

🏎️ HystrixCommand + OkHttp (Combined Example)

@Slf4j
public class ExternalServiceClient {

    private final OkHttpClient httpClient;
    private final ObjectMapper objectMapper;

    @Inject
    public ExternalServiceClient(OkHttpClient httpClient, ObjectMapper objectMapper) {
        this.httpClient = httpClient;
        this.objectMapper = objectMapper;
    }

    public BookmarkDTO getBookmark(String id) {
        // Wrap the OkHttp call in Hystrix
        return new HystrixCommand<BookmarkDTO>(
            HystrixCommandGroupKey.Factory.asKey("ExternalBookmarks")) {

            @Override
            protected BookmarkDTO run() throws IOException {
                Request request = new Request.Builder()
                    .url("https://external.api.com/bookmarks/" + id)
                    .build();

                try (Response response = httpClient.newCall(request).execute()) {
                    return objectMapper.readValue(response.body().string(), BookmarkDTO.class);
                }
            }

            @Override
            protected BookmarkDTO getFallback() {
                log.warn("Fallback triggered for bookmark: {}", id);
                BookmarkDTO fallback = new BookmarkDTO();
                fallback.setTitle("Unavailable");
                return fallback;
            }
        }.execute();
    }
}

Chapter 17: 🔲 TODO — Advanced Topics to Study Next

These topics weren't covered in this guide but are important for production-ready applications:

🔐 Security

[ ] OAuth 2.0 — Token-based authentication (JWT tokens)
[ ] HTTPS/TLS — Configure SSL in Dropwizard config.yml
[ ] LDAP Authentication — Enterprise authentication via directory services
[ ] CORS — Cross-Origin Resource Sharing configuration

📊 Observability

[ ] Distributed Tracing — Zipkin/Jaeger for tracing requests across microservices
[ ] Prometheus — Modern metrics collection (alternative to Graphite)
[ ] Grafana Dashboards — Visualizing your metrics
[ ] Structured Logging — JSON logs for log aggregation (ELK Stack)
[ ] OpenTelemetry — Unified observability standard

🏗️ Architecture

[ ] Authorization — @RolesAllowed, @PermitAll annotations
[ ] Rate Limiting — Prevent API abuse
[ ] API Versioning — Strategies for versioning your REST API
[ ] HATEOAS — Hypermedia-driven REST (links in responses)
[ ] GraphQL — Alternative to REST for flexible queries
[ ] OpenAPI/Swagger — Auto-generate API documentation

🗄️ Database

[ ] JDBI — Lighter-weight alternative to Hibernate
[ ] Connection Pooling — HikariCP configuration
[ ] Read Replicas — Routing reads vs writes
[ ] Database Sharding — Horizontal scaling

🐳 Deployment

[ ] Docker — Containerizing your Fat JAR
[ ] Kubernetes — Orchestrating containers
[ ] CI/CD — GitHub Actions / Jenkins pipelines
[ ] Blue/Green Deployment — Zero-downtime deployments
[ ] Service Discovery — Consul, Eureka

⚡ Performance

[ ] Async Resources — @Suspended and AsyncResponse in JAX-RS
[ ] Caching — Redis/Memcached with Dropwizard
[ ] Connection Timeout Tuning — Jetty thread pool settings
[ ] GraalVM Native Image — Compile to native binary for faster startup

🧪 Advanced Testing

[ ] Consumer-Driven Contract Testing — Pact framework
[ ] WireMock — Mock external HTTP services in tests
[ ] TestContainers — Real databases in Docker for integration tests
[ ] Chaos Engineering — Testing failure scenarios (Chaos Monkey)

📨 Messaging

[ ] Apache Kafka — Event streaming
[ ] RabbitMQ — Message queuing
[ ] Dropwizard Kafka Bundle — Integration

Understanding Apache Kafka: Topics, Partitions, Delivery Guarantees & Replication 🔥

Ajinkya Singh — Tue, 25 Nov 2025 19:09:21 +0000

Topic Architecture: Building Your Data Highway
Delivery Guarantees: Never Lose, Never Duplicate
Replication & ISR: Zero Downtime Architecture

Topic Architecture: Building Your Data Highway

🎯 The Foundation: Understanding Topics

Think of a Kafka topic as a multi-lane highway where your data travels. Just like how more lanes allow more cars to drive simultaneously, more partitions allow more messages to be processed in parallel.

Creating Your First Topic

/opt/kafka/bin/kafka-topics.sh --create \
  --topic order-events \
  --partitions 4 \
  --replication-factor 3 \
  --bootstrap-server localhost:9092

What just happened?

✅ Created a topic called order-events
✅ Split it into 4 parallel lanes (partitions)
✅ Made 3 copies of each partition (for safety)

🛣️ Partitions: Your Performance Dial

Scenario: E-commerce Order Processing

You run an online store with these requirements:

5,000 orders per second during peak hours
Each processing server handles 1,250 orders/second

Calculate partitions needed:

Partitions = Target Throughput ÷ Consumer Throughput
Partitions = 5,000 ÷ 1,250 = 4 partitions

The Highway Analogy Visualized

1 Partition Topic (Bottleneck):

[═══════════════════════] Single lane - all traffic stuck

4 Partition Topic (Optimal):

[═══════════════════════] Partition 0 → Consumer A
[═══════════════════════] Partition 1 → Consumer B
[═══════════════════════] Partition 2 → Consumer C
[═══════════════════════] Partition 3 → Consumer D

🔑 Message Keys: The Secret to Ordering

The Problem

Without keys, messages for the same customer could process out of order, causing chaos!

❌ Without Keys (Random Distribution):

Customer #42 creates account    → Partition 1
Customer #42 updates email      → Partition 0  ← Different partition!
Customer #42 adds phone         → Partition 2  ← Processed out of order!

✅ With Keys (Guaranteed Order):

Key: "customer-42" → hash → Partition 2

Customer #42 creates account    → Partition 2
Customer #42 updates email      → Partition 2  ← Same partition!
Customer #42 adds phone         → Partition 2  ← Perfect order!

Real-World Example: Social Media Feed

Scenario: Building an Instagram-like feed processor

When users create posts, you need to ensure each user's posts are processed in order, but different users can be processed in parallel.

Key Strategy:

User ID as Key → All posts from same user → Same partition → Guaranteed order

Result: All of Sarah's posts stay in order, but different users' posts can be processed in parallel across partitions!

Key Strategy Decision Tree

Do you need message ordering?
├─ NO  → Use NO KEY (round-robin for max throughput)
│         Perfect for: Metrics, logs, sensor data
│
└─ YES → Use MESSAGE KEY
    ├─ User activity?     → key = user_id
    ├─ Order processing?  → key = order_id
    ├─ IoT devices?       → key = device_id
    └─ Multi-tenant app?  → key = tenant_id

🛡️ Replication: Your Safety Net

Scenario: Banking Application

You're building a payment processor. One question matters most:

"If a server crashes at 2 AM, do we lose transaction records?"

Answer depends on replication factor:

Replication Factor = 1 (❌ NEVER IN PRODUCTION)

Partition 0:
└── Broker 1 (Leader) ← Single point of failure
    If Broker 1 crashes → DATA PERMANENTLY LOST

Replication Factor = 3 (✅ Production Standard)

Partition 0:
├── Broker 1 (Leader)    ← Handles all reads/writes
├── Broker 2 (Follower)  ← Backup copy
└── Broker 3 (Follower)  ← Another backup

Broker 1 crashes? → Broker 2 becomes leader → NO DATA LOSS ✓

Real Failure Scenario: Diwali Sale

Timeline of a broker failure:

11:59 PM - Diwali sale starts, 10,000 orders/second
12:03 AM - Broker 2 crashes (hardware failure)
12:03 AM - Controller detects failure (3 seconds)
12:03 AM - Broker 3 promoted to leader
12:03 AM - Orders continue processing
         - Zero orders lost ✓
         - 3-second interruption only

⚙️ Advanced Configuration: Retention & Durability

Configuration Example: Multi-Tier Data Strategy

Real-time Analytics Topic (High Volume, Short Retention)

/opt/kafka/bin/kafka-topics.sh --create \
  --topic clickstream-events \
  --partitions 16 \
  --replication-factor 3 \
  --config retention.ms=86400000 \  # 1 day
  --config min.insync.replicas=1    # Lower durability for speed

Financial Transactions Topic (Critical, Long Retention)

/opt/kafka/bin/kafka-topics.sh --create \
  --topic payment-transactions \
  --partitions 6 \
  --replication-factor 3 \
  --config retention.ms=7776000000 \  # 90 days
  --config min.insync.replicas=2      # High durability

Event Sourcing Topic (Forever, Maximum Safety)

/opt/kafka/bin/kafka-topics.sh --create \
  --topic account-events \
  --partitions 8 \
  --replication-factor 5 \
  --config retention.ms=-1 \          # Forever
  --config min.insync.replicas=3      # Maximum durability

Delivery Guarantees: Never Lose, Never Duplicate

🎭 The Central Dilemma

In distributed systems, failures are inevitable. The question is:

When your consumer crashes mid-processing, what happens to that message?

You have two choices, each with trade-offs:

📊 At Most Once: "Fire and Forget"

Definition: Process each message 0 or 1 times (never more than once)

Priority: Prevent duplicates at all costs, even if it means losing messages

The Coffee Shop Analogy

You order coffee:
1. Barista writes order ✓
2. ✅ MARKS YOUR ORDER COMPLETE (commits offset)
3. 💥 Power outage before making coffee!
4. Power returns
5. Order marked complete → You never get your coffee

Result: Order processed ZERO times (at most once)

How It Works: Automatic Offset Commit

Consumer Configuration:

enable.auto.commit=true
auto.commit.interval.ms=5000  # Commits every 5 seconds

Processing Flow:

1. Fetch message from Kafka
2. ✅ Auto-commit saves offset (marks as done)
3. Start processing message
4. If crash here → Message LOST

Timeline of Message Loss

10:00:00 - Fetch message "user_clicked_buy_button"
10:00:01 - Auto-commit saves offset 1001 ✓
10:00:02 - Start processing...
10:00:03 - 💥 APPLICATION CRASHES
10:00:04 - Restart
10:00:05 - Ask Kafka: "Where was I?"
10:00:05 - Kafka: "You were at offset 1001"
10:00:06 - Skip to offset 1001
          - Message 1000 NEVER PROCESSED

🎯 At Least Once: "Never Lose Anything"

Definition: Process each message 1 or more times (never zero times)

Priority: Ensure every message is processed, even if it means processing some twice

The Coffee Shop Analogy (Revised)

You order coffee:
1. Barista makes your coffee ✓
2. Serves it to you ✓
3. 💥 Power outage before marking order complete!
4. Power returns
5. Order NOT marked complete → Makes coffee AGAIN
6. You get TWO coffees (one extra)

Result: Order processed TWICE (at least once)

How It Works: Manual Offset Commit

Consumer Configuration:

enable.auto.commit=false  # Manual control

Processing Flow:

1. Fetch message from Kafka
2. Process message successfully ✓
3. Write to database ✓
4. Commit offset manually ✓
5. If crash before step 4 → Message REPROCESSED

Timeline of Duplicate Processing

10:00:00 - Fetch message "order_12345: charge $100"
10:00:01 - Process order successfully ✓
10:00:02 - Charge customer $100 ✓
10:00:03 - Write to database ✓
10:00:04 - 💥 APPLICATION CRASHES (before committing offset)
10:00:05 - Restart
10:00:06 - Ask Kafka: "Where was I?"
10:00:06 - Kafka: "You were at offset 1000"
10:00:07 - Fetch message 1000 AGAIN
10:00:08 - Process order AGAIN
10:00:09 - Charge customer $100 AGAIN ⚠️ (DUPLICATE!)
10:00:10 - Write to database AGAIN
10:00:11 - Commit offset successfully

🎓 Decision Matrix: Which Guarantee to Use?

                    START HERE
                        ↓
            Can you afford to lose messages?
                    ↙       ↘
                  YES        NO
                   ↓          ↓
           AT MOST ONCE   AT LEAST ONCE
                   ↓          ↓
            Use cases:   Can you handle duplicates?
            • Sensors       ↙            ↘
            • Metrics     YES            NO
            • Logs         ↓              ↓
                    Implement      Add deduplication
                    idempotency    • Transaction IDs
                    • Txn IDs      • DB constraints
                    • DB keys      • Processing log

Use Case Decision Table

Scenario	Guarantee	Reasoning
Banking transactions	At Least Once	Cannot lose money transfers
User registrations	At Least Once	Cannot lose new users
E-commerce orders	At Least Once	Cannot lose customer orders
Stock trades	At Least Once	Cannot lose trade records
IoT sensor readings	At Most Once	Losing one reading is acceptable
Application logs	At Most Once	Missing one log entry is okay
Click analytics	At Most Once	Approximate counts are fine
System metrics	At Most Once	Slightly off counts acceptable

⚙️ Configuration Comparison

At Most Once Configuration:

# Consumer config
enable.auto.commit=true
auto.commit.interval.ms=5000

At Least Once Configuration:

# Consumer config
enable.auto.commit=false  # Manual control

Replication And ISR: Zero Downtime Architecture

🏗️ The Architecture: No Single Point of Failure

The Problem Visualized

Without Replication:

Topic: customer-orders
Partition 0 → Broker 1 ← Single copy
              💥 Crashes
              ↓
          🚨 DATA LOST FOREVER
          🚨 Service DOWN

With Replication:

Topic: customer-orders
Partition 0:
├── Broker 1 (Leader)    ← Primary copy
├── Broker 2 (Follower)  ← Backup copy
└── Broker 3 (Follower)  ← Another backup

Broker 1 💥 Crashes
         ↓
Broker 2 becomes Leader (3 seconds)
         ↓
✅ NO DATA LOST
✅ Service continues

🎭 Leaders and Followers

How Replication Works

Every partition has ONE leader and multiple followers:

Partition 0 (Replication Factor = 3):

                    Producers write here
                           ↓
        ┌──────────────────────────────┐
        │  Broker 1 (LEADER)           │
        │  [msg1][msg2][msg3][msg4]    │
        └──────────────────────────────┘
                 ↓           ↓
    ┌────────────┘           └────────────┐
    ↓                                     ↓
┌──────────────────┐           ┌──────────────────┐
│ Broker 2         │           │ Broker 3         │
│ (FOLLOWER)       │           │ (FOLLOWER)       │
│ [msg1][msg2]     │           │ [msg1][msg2]     │
│ [msg3][msg4]     │           │ [msg3][msg4]     │
└──────────────────┘           └──────────────────┘
    Actively                        Actively
    replicating                     replicating

Key Rules:

All writes go through the leader
All reads go through the leader
Followers continuously pull new data from leader
Followers do NOT serve client requests

🎯 In-Sync Replicas (ISR): The Safety Net

ISR = A follower that is fully caught up and healthy

What Makes a Replica "In-Sync"?

✅ IN-SYNC REPLICA:
   • Has all committed messages
   • Actively fetching from leader
   • Not fallen behind (< replica.lag.time.max.milliseconds)
   • Ready to become leader at any moment

❌ OUT-OF-SYNC REPLICA:
   • Missing recent messages
   • Stopped fetching (network issue, crashed)
   • Fallen too far behind
   • CANNOT become leader

Real-World Analogy: Emergency Contact List

You have 3 emergency contacts:

Contact 1 (Leader): Mom
  • Always answers immediately
  • Has all current information

Contact 2 (ISR): Dad
  • Answers within seconds
  • Fully updated on family matters
  • Can step in if Mom unavailable ✓

Contact 3 (Out-of-Sync): Cousin
  • Takes hours to respond
  • Out of the loop
  • Can't be relied on in emergency ✗

🔒 Message Commitment: The Two-Phase Process

Phase 1: Leader Writes

Producer sends: "Transfer $500 from Alice to Bob"
        ↓
Leader writes to local log
        ↓
Message is "uncommitted" (not safe yet)

Phase 2: ISRs Acknowledge

Leader notifies followers:
        ↓
Follower 1 (ISR) replicates ✓
        ↓
Follower 2 (ISR) replicates ✓
        ↓
Message now "committed" (safe!)
        ↓
Producer receives ACK

Producer Configuration Impact

acks=1 (Fast but Risky):

Producer → Leader writes → ACK immediately
           (Followers not yet replicated)

Leader crashes before replication?
→ Message LOST 💥

acks=all (Slow but Safe):

Producer → Leader writes → Wait for ALL ISRs → ACK

Takes longer, but:
→ Message GUARANTEED safe ✅
→ Zero data loss

🚨 Failure Scenario: Leader Election

Real-World Example: Diwali Sale at 2 AM

Initial State:

Topic: flash-sale-orders
Partition 0:
├── Broker 1 (Leader)    ← 50,000 orders/sec
│   ISR: [1, 2, 3]
├── Broker 2 (Follower)  ← In-Sync
└── Broker 3 (Follower)  ← In-Sync

Failure Timeline:

02:00:00 - Broker 1 crashes (power supply failure)
02:00:01 - Producer attempts write → Connection refused
02:00:02 - Zookeeper detects missing heartbeat
02:00:03 - Controller initiates leader election
           Candidates: Broker 2, Broker 3 (both ISR)
02:00:03 - Broker 2 elected as new leader
02:00:04 - Controller notifies all brokers
02:00:04 - Producer auto-discovers new leader
02:00:05 - Orders resume processing ✅

Total downtime: 5 seconds
Orders lost: ZERO (because acks=all)

Why Was Broker 2 Chosen?

✅ Was in ISR (fully caught up)
✅ Had all committed messages
✅ First in preferred replica list

What If Broker 2 Was Out-of-Sync?

02:00:03 - Controller checks ISR: [1, 3]
02:00:03 - Broker 2 NOT in ISR → SKIPPED
02:00:03 - Broker 3 elected instead

⚙️ Configuration Deep Dive

1. Replication Factor (Topic Level)

# Development: Fast, no safety
/opt/kafka/bin/kafka-topics.sh --create \
  --topic dev-logs \
  --replication-factor 1  # ❌ Never in production!

# Staging: Moderate safety
/opt/kafka/bin/kafka-topics.sh --create \
  --topic staging-events \
  --replication-factor 2

# Production: Standard safety
/opt/kafka/bin/kafka-topics.sh --create \
  --topic production-orders \
  --replication-factor 3  # ✅ Industry standard

# Mission-Critical: Maximum safety
/opt/kafka/bin/kafka-topics.sh --create \
  --topic financial-transactions \
  --replication-factor 5

2. Min In-Sync Replicas (Topic Level)

Works with acks=all on producer side:

# Weak durability (fast but risky)
--config min.insync.replicas=1
# Leader only needs to write
# If leader crashes before followers replicate → DATA LOST

# Strong durability (recommended)
--config min.insync.replicas=2
# Leader + at least 1 follower must acknowledge
# Can survive 1 broker failure

# Maximum durability (mission-critical)
--config min.insync.replicas=3
# Leader + at least 2 followers must acknowledge
# Can survive 2 broker failures

3. Replica Lag Time (Broker Level)

# In server.properties
replica.lag.time.max.milliseconds=10000  # 10 seconds

# If follower doesn't fetch within 10 seconds:
# → Removed from ISR
# → Cannot become leader
# → Logs warning

Understanding Kafka Storage Architecture: How It Handles Billions of Messages⚡

Ajinkya Singh — Mon, 24 Nov 2025 19:56:31 +0000

🎯 Introduction: Beyond the Message Queue

Imagine you're running a global streaming platform processing millions of viewer events every second—play, pause, skip, like, and watch history. How do you store this tsunami of data efficiently while keeping it instantly accessible? This is where Kafka's storage architecture becomes a masterpiece of engineering.

🏗️ The Three Core Responsibilities

Every Kafka broker juggles three critical tasks simultaneously:

1. 📥 Producer Gateway

Accepts incoming streams of events from applications across your network

2. 💾 Storage Engine

Writes messages to disk durably and efficiently—this is where the magic happens

3. 📤 Consumer Server

Rapidly locates and delivers data to consumers while replicating to other brokers

📊 The Storage Hierarchy

Topic: "viewer-activity"
│
├── Partition 0 (folder: /data/viewer-activity-0/)
│   ├── 00000000000000000000.log
│   ├── 00000000000000000000.index
│   ├── 00000000000000000000.timeindex
│   ├── 00000000000000850000.log
│   ├── 00000000000000850000.index
│   ├── 00000000000000850000.timeindex
│   └── 00000000000001700000.log (active segment)
│
├── Partition 1 (folder: /data/viewer-activity-1/)
│   └── [similar segment files...]
│
└── Partition 2 (folder: /data/viewer-activity-2/)
    └── [similar segment files...]

🧩 Why Segments? The Problem with Giant Files

Imagine storing all viewer events in one massive file:

❌ A 15TB file is impossible to manage

❌ Deleting old watch history requires rewriting everything

❌ Finding "what did user watch at 3 PM yesterday" means scanning terabytes

❌ File corruption destroys all your data

The Segment Solution:

Kafka divides each partition's log into segments—smaller, manageable chunks (typically 1GB each).

🔄 When Does Kafka Create a New Segment?

A new segment rolls when either condition is met:

Condition 1: Size Threshold

Current segment reaches 1GB (default)
→ Close it, start fresh segment

Condition 2: Time Threshold

7 days have passed (default)
→ Roll to new segment regardless of size

📅 Real-World Timeline Example:

Monday 9:00 AM  → Users start watching shows
                → Messages written to segment 0

Monday 2:30 PM  → Segment 0 hits 1GB
                → Kafka creates segment 1
                → New events go to segment 1

Tuesday 3:00 PM → Segment 1 hits 1GB
                → Kafka creates segment 2 (now active)

🗂️ Segment File Anatomy

For each segment, Kafka maintains three files:

Partition Folder: viewer-activity-0/
│
├── 00000000000000850000.log        (1GB - actual messages)
├── 00000000000000850000.index      (10MB - offset lookup map)
└── 00000000000000850000.timeindex  (10MB - timestamp lookup map)

File Naming Convention

The number 00000000000000850000 is the base offset—the first message's offset in this segment.

⚡ Index Files: Kafka's Speed Secret

🎯 The Two-Step Lookup Process

When Kafka needs to find a message, it uses TWO files in sequence:

First: Check the segment file name (base offset) to find the RIGHT segment
Second: Use the .index file to find the EXACT location within that segment

Let's see this in action!

🔍 1. Offset Index (.index)

Maps message offset → byte position in the log file

How It Works:

Consumer Request: "Give me messages starting from offset 850,125"

Step 1: FIND THE RIGHT SEGMENT (using base offsets)
        Available segments:
        • 00000000000000000000.log (base: 0)
        • 00000000000000850000.log (base: 850,000) ← THIS ONE!
        • 00000000000001700000.log (base: 1,700,000)

        Offset 850,125 falls between 850,000 and 1,700,000
        → Select segment: 00000000000000850000.log

Step 2: FIND EXACT POSITION (using .index file)
        Broker loads 00000000000000850000.index into memory
        Binary search finds:
        • offset 850,100 → byte 3072
        • offset 850,150 → byte 6144

        Message 850,125 must be between bytes 3072-6144

Step 3: READ THE MESSAGE
        Read from byte 3072 in 00000000000000850000.log
        Scan forward until offset 850,125 is found

Index Structure:

┌──────────┬─────────────┐
│  Offset  │  Byte Pos   │
├──────────┼─────────────┤
│  850000  │      0      │
│  850100  │   3072      │
│  850150  │   6144      │
│  850200  │   9216      │
│   ...    │    ...      │
└──────────┴─────────────┘

Result: ✨ No scanning millions of messages—instant lookup!

📋 Visual: The Two-File Lookup System

🎯 FINDING MESSAGE BY OFFSET (850,125)

File System View:
├── 00000000000000000000.log      (offsets: 0 - 849,999)
├── 00000000000000000000.index
├── 00000000000000850000.log      (offsets: 850,000 - 1,699,999) ⭐
├── 00000000000000850000.index    ⭐ USE THIS!
├── 00000000000001700000.log      (offsets: 1,700,000+)
└── 00000000000001700000.index

Step 1: Match offset to segment name (base offset)
        850,125 is >= 850,000 and < 1,700,000
        → Open segment: 00000000000000850000

Step 2: Use that segment's .index file
        → Open: 00000000000000850000.index
        → Find byte position: 3072
        → Read from .log file at byte 3072

🔑 Key Insight: Why Two Files?

Without segment files (base offset in filename):

❌ Would need to check EVERY index file
❌ Open thousands of files to find the right one
❌ Very slow!

With segment files (base offset in filename):

✅ Filename tells you the offset range instantly
✅ Only open ONE .index file
✅ Lightning fast!

The Two-Step Magic:

Segment filename (base offset) = Coarse filter (which file?)
Index file (.index) = Fine filter (which byte position?)

⏰ 2. Time Index (.timeindex)

Maps timestamp → corresponding offset (also uses two-step lookup!)

How It Works:

Consumer Request: "Show me all viewer activity from the last hour"
                  (timestamp: 2025-11-25 13:00:00)

Step 1: FIND THE RIGHT SEGMENT (using .timeindex files)
        Check all segments' time ranges:
        • Segment 0: timestamps 10:00 - 11:59
        • Segment 850000: timestamps 12:00 - 13:59 ← THIS ONE!
        • Segment 1700000: timestamps 14:00 onwards

        Timestamp 13:00:00 falls in segment 850000

Step 2: FIND THE OFFSET (using .timeindex)
        Broker checks 00000000000000850000.timeindex
        Binary search finds:
        2025-11-25 13:00:00 → offset 850,750

Step 3: USE OFFSET INDEX (now it's a regular offset lookup!)
        Uses 00000000000000850000.index
        Finds: offset 850,750 → byte position 225,280

Step 4: READ THE MESSAGES
        Reads from byte 225,280 in .log file
        Returns all messages from that point forward

Time Index Structure:

┌──────────────────────┬──────────┐
│     Timestamp        │  Offset  │
├──────────────────────┼──────────┤
│ 2025-11-25 10:00:00  │  850000  │
│ 2025-11-25 11:30:00  │  850300  │
│ 2025-11-25 13:00:00  │  850750  │
│ 2025-11-25 14:15:00  │  851000  │
│        ...           │   ...    │
└──────────────────────┴──────────┘

🧠 The Memory Advantage

Why are index lookups so blazing fast?

Disk Storage (Slow):
├── segment.log (1GB - rarely loaded fully)
├── segment.index (10MB - loaded into RAM!)
└── segment.timeindex (10MB - loaded into RAM!)

Memory (Lightning Fast):
└── Index files cached → Microsecond lookups

Because index files are tiny (10-20MB), they fit entirely in memory:

✅ No disk reads for lookups
✅ Binary search through in-memory structures
✅ Microseconds instead of seconds

🗑️ Retention Policies: Managing Disk Space

Without cleanup, your disks would eventually overflow. Kafka automatically deletes old segments based on policies.

1. Time-Based Retention

Configuration:
log.retention.hours = 168  (7 days)

Timeline:
Day 1  → Segment 0 created (Nov 18 viewer data)
Day 2  → Segment 1 created (Nov 19 viewer data)
...
Day 8  → Segment 0 is now 7 days old
       → Kafka deletes:
         - segment-0.log
         - segment-0.index
         - segment-0.timeindex

Visual:

Before Cleanup (Day 8):
[Seg0: Nov 18] [Seg1: Nov 19] [Seg2: Nov 20] ... [Seg7: Nov 25]
      ↓ 7 days old - DELETE

After Cleanup:
[Seg1: Nov 19] [Seg2: Nov 20] ... [Seg7: Nov 25]

2. Size-Based Retention

Configuration:
log.retention.bytes = 107374182400  (100GB per partition)

Current State:
Partition total size = 98GB ✓ OK

New Segment Added:
Partition total size = 103GB ✗ EXCEEDS LIMIT

Action:
→ Kafka deletes oldest segment
→ New total = 102GB ✓

3. Combined Policies

You can use both simultaneously—Kafka deletes when either condition is met:

log.retention.hours = 168     (7 days)
log.retention.bytes = 100GB

Segment deleted if:
✓ Age > 7 days  OR
✓ Total size > 100GB

💨 Why Deletion is Lightning Fast

Segment-Level Operations

Kafka never deletes individual messages. It operates on entire segment files.

❌ Inefficient (What Kafka DOESN'T do):
Read file → Skip old messages → Write remaining → Rebuild indexes
(Hours of processing)

✅ Efficient (What Kafka DOES):
Delete segment-000.log
Delete segment-000.index  
Delete segment-000.timeindex
(Milliseconds)

Benefits:

🚀 One filesystem operation
🚀 No data rewriting
🚀 No index rebuilding
🚀 Extremely low CPU usage

🎬 Real-World Message Journey

Let's trace a viewer event through the entire system:

Step 1: Producer Sends Event

Producer → Kafka Broker

Message: {
  user_id: "viewer-789",
  action: "started_watching",
  show_id: "breaking-code-s1e3",
  timestamp: 2025-11-25 14:45:00
}

Destination:
Topic: viewer-activity
Partition: 2 (based on user_id hash)
Offset: 1,700,082 (next available)

Step 2: Write to Active Segment

Active Segment: 00000000000001700000.log
Position: Append at byte 245,760

Step 3: Update Indexes

Offset Index:
1,700,082 → byte 245,760

Time Index:
2025-11-25 14:45:00 → offset 1,700,082

Step 4: Acknowledge Producer

Broker → Producer:
"✓ Message written at offset 1,700,082"

Step 5: Consumer Reads Message

Consumer Request: "Give me messages from offset 1,700,082"

Kafka Process:
1. Which segment? → 00000000000001700000.log
2. Load offset index into memory
3. Find: offset 1,700,082 → byte 245,760
4. Read from disk starting at byte 245,760
5. Return message to consumer

🧹 Log Compaction: Latest State Only

The Use Case

Scenario: Storing user profile updates

User ID: viewer-789

Messages Over Time:
1. viewer-789 → {email: "old@email.com", plan: "basic"}
2. viewer-789 → {email: "new@email.com", plan: "basic"}
3. viewer-789 → {email: "new@email.com", plan: "premium"}

Question: Do we need messages 1 and 2?
Answer: NO! Only the latest state matters.

How Compaction Works

Configuration:
log.cleanup.policy = compact

Result:
Kafka guarantees: For each key, retain at least the last known value

Background Process:
1. Log cleaner reads old segments
2. Identifies duplicate keys
3. Keeps only latest message per key
4. Creates cleaned segments
5. Replaces old segments

💀 Tombstones: Deleting Keys

To remove a key entirely:

Send message: {key: "viewer-789", value: null}

Effect:
→ Log cleaner removes the key
→ Includes its last valid value
→ After configurable grace period

Compaction vs. Retention Comparison

Aspect	Retention (Delete)	Compaction
What's Kept	Time/size window of all messages	Latest value per key
What's Deleted	Old segments entirely	Old values for same key
Use Case	Time-series events, logs	User profiles, state management
Configuration	`log.cleanup.policy=delete`	`log.cleanup.policy=compact`
History	Limited time/size window	No history, only latest

🚀 Performance Benefits Summary

Why Kafka is Blazing Fast

Sequential Writes → Always append to end (disk-friendly)
Memory-Mapped Indexes → Entire index files in RAM
Zero-Copy Transfer → Direct disk → network socket
Batch Processing → Multiple messages written together
Segment-Level Operations → No expensive per-message work

📊 Real-World Numbers

A single Kafka broker can handle:

✅ Millions of messages per second
✅ Hundreds of MB/s throughput
✅ Sub-millisecond latencies
✅ All while maintaining durability

Learn Kafka by Doing: Build a 3-Broker Kafka Cluster with Docker Compose

Ajinkya Singh — Sun, 23 Nov 2025 11:00:00 +0000

Overview

This configuration deploys a 3-broker Kafka cluster with automatic failover, data replication, and a management UI.

Docker Compose Configuration

services:
  kafka1:
    image: apache/kafka:latest
    container_name: kafka1
    ports:
      - "9093:9093"
      - "29092:29092"
    volumes:
      - ./kafka-data/kafka1:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 1
      CLUSTER_ID: t7jxO1XIQwWtRhIzV5PE4w
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,EXTERNAL://localhost:9093
      KAFKA_LISTENERS: CONTROLLER://:9092,EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

  kafka2:
    image: apache/kafka:latest
    container_name: kafka2
    ports:
      - "9094:9093"
    volumes:
      - ./kafka-data/kafka2:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 2
      CLUSTER_ID: t7jxO1XIQwWtRhIzV5PE4w
      KAFKA_PROCESS_ROLES: broker
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:29092,EXTERNAL://localhost:9094
      KAFKA_LISTENERS: EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

  kafka3:
    image: apache/kafka:latest
    container_name: kafka3
    ports:
      - "9095:9093"
    volumes:
      - ./kafka-data/kafka3:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 3
      CLUSTER_ID: t7jxO1XIQwWtRhIzV5PE4w
      KAFKA_PROCESS_ROLES: broker
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:29092,EXTERNAL://localhost:9095
      KAFKA_LISTENERS: EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8000:8080"
    networks:
      - kafka-bridge
    environment:
      KAFKA_CLUSTERS_0_NAME: kafkamultinodecluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka1:29092,kafka2:29092,kafka3:29092"
    depends_on:
      - kafka1
      - kafka2
      - kafka3

networks:
  kafka-bridge:
    driver: bridge

Key Configuration Breakdown

Port Strategy

9093-9095: External access (your apps connect here)
29092: Internal broker communication
9092: Controller coordination
8000: Kafka UI web interface

Listeners Explained

CONTROLLER (9092) - Controllers coordinate cluster operations
INTERNAL (29092)  - Brokers communicate with each other
EXTERNAL (9093+)  - External clients connect

Critical Environment Variables

CLUSTER_ID: Must be identical across all brokers (forms the cluster)

BROKER_ID: Unique identifier per broker (1, 2, 3)

PROCESS_ROLES:

kafka1: broker,controller (both roles)
kafka2/3: broker (worker only)

CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092 (only kafka1 can be controller)

ADVERTISED_LISTENERS:

INTERNAL://kafka1:29092 - Used by other brokers
EXTERNAL://localhost:9093 - Used by external clients

Replication Factors:

KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 - Consumer offsets replicated 3x
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 - Transaction logs replicated 3x
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 - Minimum in-sync replicas (trade-off between availability and durability)

Data Persistence

./kafka-data/kafka1 → /var/lib/kafka/data (in container)
./kafka-data/kafka2 → /var/lib/kafka/data
./kafka-data/kafka3 → /var/lib/kafka/data

Data survives container restarts.

Producer Script

producer-read.sh:

#!/bin/bash
BOOTSTRAP_SERVERS="kafka2:29092"
TOPIC_NAME=$1
CONTAINER_NAME="kafka1"

if [ -z "$TOPIC_NAME" ]; then
  echo "Usage: $0 <topic-name>"
  exit 1
fi

for i in {1..10}; do
  MESSAGE="Message $i from producer"
  docker exec -it $CONTAINER_NAME bash -c "echo \"$MESSAGE\" | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server $BOOTSTRAP_SERVERS --topic $TOPIC_NAME"
  echo "Sent: $MESSAGE"
done

Usage: ./producer-read.sh customer-orders

Sends 10 messages to the specified topic through kafka2, running from kafka1 container.

Consumer Script

consumer-read.sh:

#!/bin/bash
BOOTSTRAP_SERVERS="kafka1:29092"
TOPIC_NAME=$1
CONTAINER_NAME="kafka2"

if [ -z "$TOPIC_NAME" ]; then
  echo "Usage: $0 <topic-name>"
  exit 1
fi

docker exec -it $CONTAINER_NAME bash -c "/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server $BOOTSTRAP_SERVERS --topic $TOPIC_NAME --from-beginning"

Usage: ./consumer-read.sh customer-orders

Reads all messages from the beginning, running from kafka2 container, connecting to kafka1.

Quick Start

Start cluster: docker-compose up -d
Access UI: http://localhost:8000
Send messages: ./producer-read.sh my-topic
Read messages: ./consumer-read.sh my-topic
Stop cluster: docker-compose down

Architecture Summary

┌─────────────────────────────────────────────┐
│  kafka1 (broker + controller)               │
│  Ports: 9093 (external), 29092 (internal)   │
└─────────────────────────────────────────────┘
                    ↕
┌─────────────────────────────────────────────┐
│  kafka2 (broker)                            │
│  Ports: 9094 (external), 29092 (internal)   │
└─────────────────────────────────────────────┘
                    ↕
┌─────────────────────────────────────────────┐
│  kafka3 (broker)                            │
│  Ports: 9095 (external), 29092 (internal)   │
└─────────────────────────────────────────────┘
                    ↕
┌─────────────────────────────────────────────┐
│  kafka-ui (management interface)            │
│  Port: 8000                                 │
└─────────────────────────────────────────────┘

All brokers share the same CLUSTER_ID and communicate on the kafka-bridge network.

DEV Community: Ajinkya Singh

Multi-Core Cache Coherency: The Invisible Lock You Never Wrote

Recap — and the question this article finally answers

The new problem: private caches, shared memory

The four states every cached line is tagged with (MESI)

What actually crosses the wires

A cost ladder — not every core-to-core interaction is equal

The key insight: invalidation happens per line, not per variable

Wait — doesn't this mean coherency already prevents races?

Glossary added this part

Takeaways

The 64-Byte Truck: Cache Lines Explained

Recap

What actually happens when you read one int

Why 64 bytes specifically — two independent reasons

Reason 1: the physical width of the connection

Reason 2: spatial locality says it's worth it

Cache lines are aligned, not centered on what you asked for

How much fits in one line

Glossary added this part

Takeaways

The Hardware Memory Hierarchy: From L1 Caches to Main RAM

Recap

The problem caches solve

The hierarchy, level by level

Why a pyramid, not one big fast memory

How the hierarchy is actually used

Why caching works at all: locality

Glossary added this part

Takeaways

How CPUs and Memory Actually Work: Bytes, Addresses, and Latency

Recap — the question this series has been avoiding

A computer, stripped down to two things talking to each other

Cycles — the CPU's metronome

Bytes and addresses — a variable is just a name for a location

Fast vs. slow — the ordering that everything else hangs on

Cores — why "more than one CPU" changes everything

Takeaways

C++ Multithreading: Why -O2 Deletes Your Loops and How std::atomic Fixes It

Recap

The setup: a flag, and a thread that waits on it

The problem: the compiler doesn't just get this wrong — it gets it wrong differently at every optimization level

Why: the compiler is allowed to assume this bug doesn't exist

std::atomic actually does two different jobs — don't blur them together

Back to ready — this article only ever needed Job 2

The fix: std::atomic<bool>

Why this matters more than the Part 1 bug did

Takeaways

Understanding C++ Threads: Data Races Explained Down to the Assembly

Notes & References

The setup

The problem: the obvious multithreaded version is wrong

Why: cnt += x is not one instruction

Solution attempt 1: lock the notebook — a mutex

Solution attempt 2: don't share the notebook at all

The full picture

Takeaways

Kubernetes Core Workloads: Everything You Need to Know

The Quick Reference: Which Workload Do You Need?

Pods: The Atomic Unit

The Most Important Thing About Pods

Two Probes You Can't Skip

Pod Status Lifecycle

Quick Debugging Playbook

🧪 Practice — Pods

Namespaces: Logical Isolation

What Namespaces Give You

One Critical Misconception

🧪 Practice — Namespaces

Labels, Selectors & Annotations

How a Service Uses Labels

Set-Based Selectors

🧪 Practice — Labels & Selectors

ReplicaSets: Self-Healing

Why You Almost Never Create ReplicaSets Directly

🧪 Practice — ReplicaSets

Deployments: The Standard Way to Run Apps

How a Rolling Update Actually Works

Production Deployment YAML

The Commands You'll Use Daily

What actually happens when you read one `int`

`std::atomic` actually does two different jobs — don't blur them together

Back to `ready` — this article only ever needed Job 2

The fix: `std::atomic<bool>`

Why: `cnt += x` is not one instruction

📄 The `pom.xml` — Your Project's Heart

📄 The `config.yml` File