DEV Community: Ajinkya Singh

Pull, Don't Push: Fixing Load Imbalance with a Shared Task Queue

Ajinkya Singh — Thu, 17 Sep 2026 20:00:50 +0000

Sources this series builds on: Chili's (ChiliTomatoNoodle) multithreading
playlist,
CMU 15-213 (Computer Systems: A Programmer's Perspective), various CppCon talks,
and LLMs for restructuring and sanity-checking. Everything here is my own code,
compiled and run — not just transcribed.

Recap

Part 8 measured the actual cost of a bad assumption: split a chunk of tasks
into 4 equal-count slices, hand one slice to each worker, and hope the
work inside those slices is roughly equal too. In the Stacked layout it
wasn't — one worker got every heavy task, the other three got none, and
three of four workers sat idle ~85% of the time waiting on the one that
didn't. The fix isn't a smarter way to pre-split the pile. It's to stop
pre-splitting at all.

The actual problem with fixed slicing

Every version in Part 8 decided, up front, before a single task ran, exactly
which tasks each worker would get. That decision was made with zero
information about how expensive any individual task actually was. Once a
worker's slice is heavy, there's no way to hand some of that work to an
idle neighbor mid-chunk — the assignment was locked in before anyone knew
it was a bad one.

flowchart LR
    subgraph Push["Push model (Part 8)"]
        direction LR
        P["Pile of tasks"] --> PS["Pre-split into 4<br/>fixed slices, up front"]
        PS --> PW1["Worker 1's slice"]
        PS --> PW2["Worker 2's slice"]
        PS --> PW3["Worker 3's slice"]
        PS --> PW4["Worker 4's slice<br/>(happens to be heavy)"]
    end

    style PW4 fill:#FAECE7,stroke:#993C1D,color:#4A1B0C

The fix: instead of deciding assignments in advance, let each worker pull
the next task from a shared pile the moment it's free. A worker that got
lucky with cheap tasks finishes fast and immediately comes back for more.
A worker stuck on an expensive task just takes longer on that one task —
but it isn't sitting on a whole pre-assigned pile of them anymore, because
there never was one.

flowchart LR
    subgraph Pull["Pull model (this article)"]
        direction LR
        Q["Shared task queue<br/>(one pile, shared cursor)"] --> QW1["Worker 1 pulls<br/>when free"]
        Q --> QW2["Worker 2 pulls<br/>when free"]
        Q --> QW3["Worker 3 pulls<br/>when free"]
        Q --> QW4["Worker 4 pulls<br/>when free"]
    end

    style Q fill:#E1F5EE,stroke:#0F6E56,color:#04342C

Nobody decides in advance who gets which task. The pile naturally drains
faster through whichever workers happen to be free at any given moment.

The real implementation: `tq::Master` and `tq::Worker`

The shared cursor

class Master {
private:
    std::span<Task> chunk;
    std::condition_variable cv;
    std::mutex mtx;
    std::atomic<int> idx;
    int doneCount = 0;
public:
    Master() {};
    void setChunk(std::span<Task> c) {
        {
            std::lock_guard<std::mutex> lock(mtx);
            chunk = c;
            idx = 0;
        }
    }

    void setDone() {
        bool notify = false;
        {
            std::lock_guard<std::mutex> lock(mtx);
            ++doneCount;
            notify = (doneCount == WORKER_COUNT);
        }
        if (notify) cv.notify_one();
    }

    const Task* getTask() {
        int currentIdx = idx++;
        if (currentIdx >= CHUNK_SIZE) return nullptr;
        return &chunk[currentIdx];
    }

    void wait_for_all() {
        std::unique_lock<std::mutex> lock(mtx);
        cv.wait(lock, [this]() ->  bool { return doneCount == WORKER_COUNT; });
        doneCount = 0;
    }
};

getTask() is the entire mechanism. idx is a std::atomic<int>, and
idx++ on it is a single locked hardware instruction (Part 6) — every
worker calling getTask() at the same moment gets a genuinely unique,
non-overlapping index back, with no mutex around the hot path at all.
Whichever worker happens to call it first gets index 0; the next caller,
whoever that is, gets index 1 — assignment order is decided by whichever
worker actually shows up asking, not by anything fixed in advance. Once
idx passes CHUNK_SIZE, every subsequent caller gets nullptr, which is
how a worker knows the pile is empty and it's time to stop.

The mutex/condition_variable pair here is doing a completely different
job from idx — it's not protecting the task dispatch itself, it's used
for setChunk (loading a new chunk between rounds) and wait_for_all
(blocking the main thread until every worker reports done). The hot,
frequently-called path — getTask() — never touches the mutex at all.

The worker loop

class Worker {
private:
    Master* master;
    bool working;
    bool dying;
    std::condition_variable cv;
    std::mutex mtx;
    std::jthread thread;
    unsigned int accumulation;
    int heavyProcessed = 0;
    float timeSpentPerWorker = 0.;
    Timer* t;

private:
    void Run_() {
        std::unique_lock<std::mutex> lock(mtx);
        while (true) {
            cv.wait(lock, [this]() -> bool { return working || dying; });
            if (dying) {
                return;
            }
            t->start();
            while (const Task* task = master->getTask()) {
                accumulation += task->process();
                heavyProcessed += task->heavy ? 1 : 0;
            }
            timeSpentPerWorker = t->stop("Worker Processing Time", false);

            working = false;
            master->setDone();
        }
    }
public:
    Worker(Master* master, Timer* t) : master(master), working(false), dying(false), accumulation(0), thread(&Worker::Run_, this), t(t) {};
    void startWorking() {
        heavyProcessed = 0;
        {
            std::unique_lock<std::mutex> lock(mtx);
            working = true;
        }
        cv.notify_one();
    }
    void Kill()
    {
        {
            std::lock_guard<std::mutex> lg(mtx);
            dying = true;
        }
        cv.notify_one();
    }

    int getHeavyCount() const { return heavyProcessed; }
    float getTimeSpentPerWorker() const { return timeSpentPerWorker; }

    double GetResult() const { return accumulation; }
};

The inner loop is the whole idea in one line:

while (const Task* task = master->getTask()) {
    accumulation += task->process();
    heavyProcessed += task->heavy ? 1 : 0;
}

No pre-assigned span, no fixed slice — just "keep asking for the next task
until there isn't one." A worker that burns through five cheap tasks in the
time another worker spends on one heavy task will simply have called
getTask() five more times. The queue doesn't know or care about heavy
at all; it never needs to, because it isn't trying to predict cost, it's
just handing out the next item to whoever's free.

Wiring it up

void Thread::simpleMultiThreadTaskQueue() {
    std::vector<tq::Chunk> chunks = generateStackedDataSeti();
    t.start();
    tq::Master master;
    std::vector<std::unique_ptr<tq::Worker>> workers;
    std::vector<ChunkTimingInfo> chunkTimings;
    chunkTimings.reserve(CHUNK_COUNT);
    for (int i = 0; i < WORKER_COUNT; i++) {
        workers.push_back(std::make_unique<tq::Worker>(&master, &t));
    }

    for (auto& chunk : chunks) {
        t.start();
        master.setChunk(chunk);
        for (auto& worker : workers) {
            worker->startWorking();
        }
        master.wait_for_all();
        float totalTime = t.stop("Chunk Processing Time", false);

        ChunkTimingInfo cTiming;
        for (int i = 0; i < WORKER_COUNT; i++) {
            cTiming.timeSpentPerWorker[i] = workers[i]->getTimeSpentPerWorker();
            cTiming.heavyCountPerWorker[i] = workers[i]->getHeavyCount();
        }
        cTiming.totalChunkTime = totalTime;
        chunkTimings.push_back(cTiming);
    }
    unsigned int answer = 0.0;
    for (auto& w : workers) answer += w->GetResult();
    std::cout << "Total sum: " << answer << "\n";

    for (auto& w : workers) w->Kill();
    workers.clear();
    t.stop("Multi Thread Task Queue", true);
    // ... CSV output, same ChunkTimingInfo format as Part 8
}

Deliberately run against generateStackedDataSeti() — the exact worst-case
layout from Part 8, all the heavy tasks clustered together. If the pull
model actually fixes the imbalance, this is the hardest dataset it could be
asked to prove that on. Same ChunkTimingInfo struct, same per-worker
timing and heavy-count tracking as Part 8's WorkerController, so the
results are directly comparable, chunk for chunk.

Why the atomic cursor specifically, not a mutex-protected queue

It would also work to protect idx (or a real std::queue) with a plain
std::mutex, locked and unlocked on every getTask() call. The reason
std::atomic<int> is used instead ties straight back to Part 6: idx++
compiles to a single lock-prefixed instruction. A mutex, by contrast,
means every worker blocks and potentially sleeps/wakes through the OS
scheduler just to grab the next index — real overhead, paid on every single
task dispatch, for work that doesn't need anything close to a full mutex's
guarantees. This is genuinely the shape of problem atomics exist for: a
tiny, hot, frequently-contended piece of shared state, where the operation
needed is simple enough that a locked hardware instruction covers it
completely.

Results — same worst-case dataset, 100 chunks

Part 8's Stacked layout — the exact same 167-heavy-task cluster used
here — produced 63.68% idle time with fixed slicing. Run the identical
dataset through the task queue instead:

Approach	Avg chunk time	Idle as % of available worker-time
Fixed slicing (Part 8, Stacked)	0.007561s	63.68%
Pull-based queue (this article)	0.002982s	0.67%

Same total work, same worker count, same worst-case task arrangement.
Idle time drops from 63.68% to 0.67% — and chunk time itself drops by
2.54x, purely from letting workers self-balance instead of guessing
their assignments up front.

The per-worker breakdown shows exactly why. Despite the queue having no
concept of heavy whatsoever, every worker ends up with almost the same
number of heavy tasks:

Worker	Avg time	Avg idle	Idle %	Avg heavy tasks
0	0.002962s	0.000020s	0.67%	42.0
1	0.002962s	0.000020s	0.68%	41.9
2	0.002962s	0.000020s	0.67%	42.0
3	0.002962s	0.000020s	0.66%	41.0

Compare that to Part 8's Stacked breakdown: one worker with all 167 heavy
tasks, three with zero. Here the same 167 heavy tasks split almost exactly
41–42 per worker, without the queue ever checking a single task's heavy
flag before handing it out. The balance isn't the result of a smarter
scheduling decision — it's a side effect of pulling one task at a time
instead of committing to a block of them in advance. A worker that
happens to draw several light tasks in a row simply calls getTask() more
often in the same span of time, and naturally ends up drawing into the
heavy cluster more too, since the cluster is still sitting in the same
shared pile everyone is pulling from.

Takeaways

Fixed, pre-decided slicing (Part 8) locks in an assignment before anyone knows if it's a good one. A pull-based queue never has to guess — workers self-balance by simply asking for more work exactly when they're free.
On the exact same worst-case dataset, idle time dropped from 63.68% (fixed slicing) to 0.67% (pull-based queue) — and chunk time itself dropped 2.54x.
The queue never checks a task's heavy flag, yet every worker ended up with almost the same heavy-task count (~41–42 each, versus 167-vs-0 with fixed slicing). The balance is a side effect of pulling one task at a time, not a scheduling decision anyone made on purpose.
The mechanism is a single std::atomic<int> cursor, incremented with a locked instruction on every request — not a queue data structure guarded by a mutex. Simpler, and cheaper on the hot path.
Tested against the exact Stacked layout that produced ~85% idle time on individual workers in Part 8 — the hardest case available, not a favorable one picked to make the fix look better than it is.

Load Balancing in C++ Multithreading: Why Equal Splits Aren't Equal Work

Ajinkya Singh — Mon, 07 Sep 2026 19:52:54 +0000

Recap

Parts 3 through 7 were entirely about where data lives — bytes, cache
lines, coherency, false sharing. All of that assumed the actual work being
handed to each thread was already fair: four threads, four equal chunks,
done. This article breaks that assumption. Splitting a job into four
equal-sized pieces by count is not the same as splitting it into four
equal-sized pieces by actual work, and the gap between those two ideas is
a completely different kind of slowdown than anything the hardware track
covered — this one lives entirely in how the job is divided, not in where
its variables sit in memory.

What we're actually studying here: unbalanced load

Strip away the specific code for a moment and name the general shape of
the problem, because it's one you'll recognize outside this series too:
you have N workers and a pile of work, and you split that pile into N
equal-count groups — one per worker. That's the intuitive, almost
automatic way to divide work. It's also only correct under one silent
assumption: that every item in the pile costs roughly the same to process.

The moment that assumption breaks — some items are cheap, some are
expensive, and the expensive ones aren't spread out evenly — equal-count
splitting quietly stops meaning equal-work splitting. Some workers finish
early and sit idle. Others are still grinding through a disproportionate
share of the expensive items. The total work didn't change, and the number
of workers didn't change — only how unevenly the cost of that work was
distributed across them.

That's what "unbalanced load" means as a general term, and it's the exact
case this article measures, concretely, with real code and real numbers:
same tasks, same worker count, three different arrangements of where the
expensive tasks happen to land.

The task: not all work is the same size

class Task {
public:
    double val;
    bool heavy;

    unsigned int process() const {
        size_t iterations = heavy ? HEAVY_IT : LIGHT_IT;
        double intermediate = val;

        for (auto i = 0; i < iterations; i++) {
            intermediate = static_cast<double>(static_cast<unsigned int>(std::abs(std::sin(std::cos(intermediate)) * 10'000'000)) % 100'000) / 10'000;
        }
        return static_cast<unsigned int>(std::exp(intermediate));
    }
};

using Chunk = std::array<Task, CHUNK_SIZE>;

Every Task carries a heavy flag. A "heavy" task runs HEAVY_IT
iterations of the inner loop; a "light" task runs LIGHT_IT — an order of
magnitude fewer. Same function, same code path, wildly different cost
depending on that one boolean. Split CHUNK_SIZE tasks into 4 equal-count
groups of CHUNK_SIZE / 4 each, and you've guaranteed each thread gets the
same number of tasks — you've guaranteed nothing about how much actual
work landed in each group.

Three ways to lay out the same tasks

std::vector<Chunk> Thread::generateRandomDataSet() {
    std::vector<Chunk> chunks;
    chunks.reserve(CHUNK_COUNT);

    std::mt19937 rnd(2828);
    std::uniform_real_distribution<double> v_dist{ 0, std::numbers::pi };
    std::bernoulli_distribution h_dist{ HEAVY_PROBABILITY };

    for (size_t c = 0; c<CHUNK_COUNT; c++) {
        Chunk chunk;
        for (auto& task : chunk) {
            task.val = v_dist(rnd);
            task.heavy = h_dist(rnd);
        }
        chunks.push_back(chunk);
    }
    return chunks;
}

std::vector<Chunk> Thread::generateEvenlyDataSet() {
    std::vector<Chunk> chunks;
    chunks.reserve(CHUNK_COUNT);

    std::mt19937 rnd(2828);
    std::uniform_real_distribution<double> v_dist{ 0, std::numbers::pi };
    int nth = static_cast<int>(1.0 / HEAVY_PROBABILITY);
    for (size_t c = 0; c < CHUNK_COUNT; c++) {
        Chunk chunk;
        for (auto it = 0; it<chunk.size(); it++) {
            chunk[it].val = v_dist(rnd);
            chunk[it].heavy = (it % nth == 0);
        }
        chunks.push_back(chunk);
    }
    return chunks;
}

std::vector<Chunk> Thread::generateStackedDataSet() {
    std::vector<Chunk> chunks = generateEvenlyDataSet();
    for (auto& chunk : chunks) {
        std::ranges::partition(chunk, [](const Task& t) {return t.heavy; });
    }
    return chunks;
}

Three datasets, same total number of heavy and light tasks in each chunk —
only the arrangement changes:

Random — each task independently has a HEAVY_PROBABILITY chance of being heavy. Heavy tasks land wherever the random draw puts them.
Evenly — every nth task is heavy, spaced out deterministically through the chunk (nth = 1 / HEAVY_PROBABILITY).
Stacked — take the evenly-spaced layout and std::ranges::partition it, sorting all the heavy tasks to one side. This is the worst case on purpose: every heavy task now clusters together instead of being spread out.

If you then split each Chunk into 4 equal-count slices — CHUNK_SIZE / 4
tasks per thread — the Stacked layout guarantees one or two threads get
almost every heavy task in the chunk, while the others get almost none.

Measuring the imbalance, not just guessing at it

class WorkerController {
    // ...
    float timeSpendPerWorker = -1.f;
    size_t heavyCount = 0;
    // ...
    void processTasks() {
        heavyCount = 0;
        for (const auto& task : input) {
            *output += task.process();
            heavyCount += task.heavy ? 1 : 0;
        }
    }
public:
    float getTimeSpentPerWorker() const { return timeSpendPerWorker; }
    size_t getHeavyCount() const { return heavyCount; }
};

Each WorkerController tracks two things per chunk it processes: how long
it actually spent, and how many heavy tasks it happened to get. Both get
written out per chunk:

struct ChunkTimingInfo {
    std::array<float, WORKER_COUNT> timeSpentPerWorker;
    std::array<size_t, WORKER_COUNT> heavyCountPerWorker;
    float totalChunkTime;
};

for (auto& chunk : chunks) {
    t.start();
    for (int j = 0; j < 4; j++) {
        std::span<Task> batch(chunk.begin() + j * (CHUNK_SIZE / 4), chunk.begin() + (j + 1) * (CHUNK_SIZE / 4));
        workers[j]->setJob(batch, &outputs[j]);
    }
    master.wait_for_all();
    float chunkTime = t.stop("Chunk Processing Time", false);
    ChunkTimingInfo info;
    for (int i = 0; i < 4; i++) {
        info.timeSpentPerWorker[i] = workers[i]->getTimeSpentPerWorker();
        info.heavyCountPerWorker[i] = workers[i]->getHeavyCount();
    }
    info.totalChunkTime = chunkTime;
    chunkTimings.push_back(info);
}

Every chunk gets split into 4 equal-count slices, handed to 4 persistent
workers, and master.wait_for_all() blocks until every worker reports done
— meaning the total chunk time is bounded by whichever worker finishes
last. The per-worker CSV this produces is the actual measurement of the
imbalance, not a guess:

std::ofstream outFile(std::format("chunk_timings_{}.csv", s), std::ios_base::trunc);
for (int i = 0; i < 4; i++) {
    outFile << std::format("time_worker_{0:},idle_time_{0:},heavy_{0:},", i);
}
outFile << "total_chunk_time,total_idle,total_heavy\n";

for (const auto& info : chunkTimings) {
    float totalIdleTime = 0.f;
    int totalHeavyCount = 0;
    for (int i = 0; i < 4; i++) {
        float idleTime = info.totalChunkTime - info.timeSpentPerWorker[i];
        outFile << std::format("{},{},{},", info.timeSpentPerWorker[i], idleTime, info.heavyCountPerWorker[i]);
        totalIdleTime += idleTime;
        totalHeavyCount += info.heavyCountPerWorker[i];
    }
    outFile << std::format("{},{},{}\n", info.totalChunkTime, totalIdleTime, totalHeavyCount);
}

idleTime here is the key number: for each worker, it's simply how much
time it spent doing nothing while waiting for the slowest worker to catch
up. A perfectly balanced chunk has every worker's idle time near zero — all
four finish at roughly the same moment. An imbalanced chunk has some
workers idle for a real fraction of the total chunk time, because they ran
out of tasks long before the worker holding all the heavy ones did.

Why this is invisible if you only look at total time

Here's the trap: summing up "total time" across a whole run can look
completely fine even when individual chunks are badly imbalanced, because
imbalance in one chunk doesn't change the total amount of work done —
it only changes how that work was distributed across the 4 workers during
that specific chunk. You have to look at idle time per worker, per chunk,
to actually see the problem. Aggregate throughput numbers can hide it
entirely.

flowchart TD
    C["One chunk, 4 equal-count slices"] --> W1["Worker 1: mostly light tasks<br/>finishes early, sits idle"]
    C --> W2["Worker 2: mostly light tasks<br/>finishes early, sits idle"]
    C --> W3["Worker 3: mostly light tasks<br/>finishes early, sits idle"]
    C --> W4["Worker 4: gets the heavy cluster<br/>still working"]
    W1 & W2 & W3 & W4 --> Wait["Chunk isn't done until<br/>Worker 4 finishes"]

    style W4 fill:#FAECE7,stroke:#993C1D,color:#4A1B0C
    style Wait fill:#FAEEDA,stroke:#854F0B,color:#412402

Splitting by count guarantees each worker gets the same number of tasks.
It says nothing about whether those tasks cost the same to run, and once
they don't, the whole chunk's completion time is bound to whichever worker
got unlucky — not to the average.

Results — 100 chunks, each layout, same machine

Averaged across 100 chunks per layout, computed directly from the CSV this
code produces:

Layout	Avg chunk time	Avg total idle (sum of 4 workers)	Idle as % of total worker-time available
Evenly	0.0162s	0.00937s	14.5%
Random	0.01657s	0.01401s	21.1%
Stacked	0.00756s	0.01926s	63.7%

That last column is the number that matters: it's total idle time across
all 4 workers, divided by the total worker-time that was actually
available (4 × chunk time) — the fraction of all available work-seconds
that were spent doing nothing.

Evenly is the best case, by design: heavy tasks are spaced out so every
nth task lands in a different worker's slice, and every worker's slice
ends up with almost exactly 100 heavy tasks. Per-worker idle time barely
varies — 13.6% to 15.6% across the four:

Worker	Avg time	Avg idle	Idle %	Avg heavy tasks
0	0.01398s	0.00222s	13.7%	100.0
1	0.01379s	0.00241s	14.9%	100.0
2	0.01367s	0.00253s	15.6%	100.0
3	0.01399s	0.00221s	13.6%	100.0

Random gets a real, measurable penalty even though heavy tasks are
still spread out roughly evenly on average — each worker ends up with
close to 100 heavy tasks too (99.4 to 100.4). The difference is that
"close to equal on average across 100 chunks" doesn't mean "equal in any
one specific chunk" — the per-chunk variance in exactly how many heavy
tasks land in each quarter is enough to push idle time up to 20.4%–22.1%
per worker, noticeably worse than Evenly's tight spread despite very
similar heavy-task counts:

Worker	Avg time	Avg idle	Idle %	Avg heavy tasks
0	0.01318s	0.00338s	20.4%	100.4
1	0.01305s	0.00352s	21.2%	99.8
2	0.01291s	0.00366s	22.1%	99.4
3	0.01311s	0.00345s	20.8%	99.5

Stacked is where the effect stops being a percentage-point difference
and becomes a completely different shape of result. One worker gets
every single heavy task in the chunk. The other three get none at all:

Worker	Avg time	Avg idle	Idle %	Avg heavy tasks
0	0.00752s	0.00004s	0.5%	167.0
1	0.00115s	0.00641s	84.8%	0.0
2	0.00114s	0.00642s	84.9%	0.0
3	0.00118s	0.00639s	84.5%	0.0

Three of the four workers spend roughly 85% of every chunk sitting
completely idle, waiting on the one worker that got the entire heavy
cluster. This is the concrete version of the diagram above — not a
hypothetical, the actual measured outcome once heavy tasks are clustered
instead of spread out.

Worth being precise about what changed and what didn't: the total amount
of heavy work in the Stacked dataset differs from Evenly's in this specific
run (167 vs 400 heavy tasks per chunk) — that's a difference in the
underlying dataset, not something this comparison is claiming to control
for. What the per-worker breakdown isolates cleanly, independent of that,
is the shape of the imbalance: concentrate the same category of
expensive tasks together instead of spreading them out, and idle time
stops being a modest tax and becomes the dominant cost of the chunk.

Takeaways

Splitting a job into N equal-count pieces only equals "balanced work" if every piece of work costs the same. The moment cost varies — a heavy flag, variable-length data, anything — count-based splitting stops guaranteeing anything about actual balance.
Layout matters as much as proportion. Measured across 100 real chunks, idle time went from 14.5% (Evenly) to 21.1% (Random) to 63.7% (Stacked) — same total heavy-task category, only its arrangement changed.
Even random placement, which balances heavy tasks close to equally on average, still costs more than deliberate spacing — average-case balance across many chunks isn't the same as balance within any single chunk.
In the worst measured case (Stacked), three of four workers sat idle ~85% of the time, waiting on the one worker that inherited the entire heavy-task cluster — a chunk's completion time is bound to its slowest worker, not its average.
Aggregate/total timing numbers can hide this problem completely. Only measuring idle time per worker, per chunk, actually reveals it.

False Sharing, Measured: Two Threads, Zero Shared Variables, Still Slow

Ajinkya Singh — Sat, 05 Sep 2026 10:56:13 +0000

Recap

Part 6 ended on a genuinely strange claim: two threads can share zero
variables in their own program logic — no mutex, no atomic touching the
same address, nothing that looks like synchronization in the code at all —
and still slow each other down, purely because the coherency protocol
invalidates an entire 64-byte cache line whenever any part of it is
written. This article makes that claim concrete, with real code from this
series.

The setup: separate variables, same benchmark

Back in Part 1, simpleMultiThreadStoragePerThread() gave each thread its
own int in a std::vector<int> cnts(4, 0) to avoid the shared-cnt race
entirely:

void Thread::simpleMultiThreadStoragePerThread() {
    t.start();
    int sum = 0;
    std::vector<int> cnts(4, 0);
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnts[i])));
    }

    for (auto& t : threads) {
        if (t.joinable()) {
            t.join();
        }
    }

    for (auto& cnt : cnts) {
        sum += cnt;
    }
    t.stop("Multi Thread Storage Per Thread", true);
    std::cout << "cnt: " << sum << "\n";
}

No two threads ever write the same int. By every rule covered in Parts 1
and 2, this is correct and race-free. What Part 1 didn't check is where
those four ints actually live in memory. A std::vector<int> packs its
elements contiguously — cnts[0], cnts[1], cnts[2], cnts[3] sit right
next to each other, 4 bytes apart. A 64-byte cache line holds 16 ints.
All four of cnts' elements — every thread's private counter — almost
certainly land in the same single cache line.

By Part 6's rule, that line is now a hot contention point: every write by
Thread 0 to cnts[0] invalidates the whole line for every core holding it
— including the cores running Threads 1, 2, and 3, which never touch
cnts[0] at all.

The simple fix already sitting in this series' code: `alignas(64)`

struct AlignedCount {
    alignas(64) int cnt = 0;
};

void Thread::simpleMultiThreadStoragePerThreadWithAligned() {
    t.start();
    int sum = 0;
    std::vector<AlignedCount> cnts(4);
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnts[i].cnt)));
    }

    for (auto& t : threads) {
        if (t.joinable()) {
            t.join();
        }
    }

    for (auto& cnt : cnts) {
        sum += cnt.cnt;
    }
    t.stop("Multi Thread Storage Per Thread With Aligned", true);
    std::cout << "cnt: " << sum << "\n";
}

alignas(64) forces the compiler to place each AlignedCount on its own
64-byte boundary — meaning cnts[0].cnt, cnts[1].cnt, cnts[2].cnt, and
cnts[3].cnt each end up in a different cache line, with no other
thread's data anywhere nearby. Nothing about the algorithm changed. Nothing
about correctness changed — this was already race-free. The only thing
that changed is memory layout, and by Part 6's cost ladder, that's the
entire difference between "every write is the expensive dirty-write-while-
another-core-holds-it case" and "every write after the first is free,
purely local."

The controlled experiment: `prac::FalseSharing`

The per-thread-storage benchmark above mixes false sharing in with 100
million iterations of sin(cos(x)) math, which makes the effect hard to
isolate. prac::FalseSharing strips everything else away and measures
only the cache-line effect:

struct SharedLine {
    alignas(8) std::atomic<uint64_t> a{ 0 };
    alignas(8) std::atomic<uint64_t> b{ 0 };
};

struct PaddedLine {
    alignas(64) std::atomic<uint64_t> a{ 0 };
    char padding[64 - sizeof(std::atomic<uint64_t>)];
    alignas(64) std::atomic<uint64_t> b{ 0 };
    char padding2[64 - sizeof(std::atomic<uint64_t>)];
};

class FalseSharing {
    template <typename T>
    double runTest(const char* label, Timer t) {
        T data;
        auto worker1 = [&data]() {
            for (uint64_t i = 0; i < kIters; ++i) {
                data.a.fetch_add(1, std::memory_order_relaxed);
            }
        };
        std::function<void()> worker2 = [&data]() {
            for (uint64_t i = 0; i < kIters; ++i) {
                data.b.fetch_add(1, std::memory_order_relaxed);
            }
        };

        t.start();
        std::thread t1(worker1);
        std::thread t2(worker2);
        t1.join();
        t2.join();
        double count = t.stop("", false);
        double m_incs = (2.0 * kIters) / (count * 1e6);

        std::cout << label << " - Time: " << count << " seconds, Throughput: " << m_incs << " M increments/sec\n";

        if (data.a.load(std::memory_order_relaxed) != kIters || data.b.load(std::memory_order_relaxed) != kIters) {
            std::cerr << "Error: Incorrect final values!\n";
        }
        return count;
    }

public:
    void run(Timer t) {
        double sec1 = runTest<SharedLine>("SharedLine", t);
        double sec2 = runTest<PaddedLine>("Padded", t);
        std::cout << "Speedup: " << sec1 / sec2 << "x\n";
    }
};

Two threads. Each does kIters relaxed fetch_adds — on its own private
atomic. Thread 1 only ever touches data.a. Thread 2 only ever touches
data.b. By construction, there is no data race here at all, in either
struct — the only thing that changes between SharedLine and PaddedLine
is memory layout:

SharedLine — a and b are 8 bytes apart, alignas(8). Both almost certainly land in the same 64-byte cache line. Every fetch_add on a is a lock-prefixed instruction (Part 6) that has to win exclusive ownership of that line — and since Thread 2 is constantly writing b in the same line, ownership is bouncing back and forth on essentially every operation. This is the expensive "write while the other core holds it dirty" case from Part 6's cost ladder, over and over.
PaddedLine — a and b are each forced onto their own 64-byte line via alignas(64) plus explicit padding to fill out the rest of the line. After the very first access, each thread holds its own line Modified permanently. Every subsequent fetch_add is a local hit — the cheapest case on Part 6's ladder, zero interconnect traffic.

Results — run back to back, same machine

[Multi Thread Storage Per Thread]: 1.17731 seconds
cnt: 266666664

[Multi Thread Storage Per Thread With Aligned]: 1.24702 seconds
cnt: 266666664

SharedLine - Time: 4.58649 seconds, Throughput: 87.2127 M increments/sec
Padded - Time: 1.80964 seconds, Throughput: 221.038 M increments/sec

Version	Time	Correct?
`simpleMultiThreadStoragePerThread()`	1.17731s	✅
`simpleMultiThreadStoragePerThreadWithAligned()`	1.24702s	✅

Version	Time	Throughput	Notes
`SharedLine`	4.58649s	87.2 M incs/sec	`a`/`b` share one cache line
`PaddedLine`	1.80964s	221.0 M incs/sec	`a`/`b` on separate cache lines

prac::FalseSharing shows the effect exactly as predicted — padding gives
a ~2.5x speedup, almost identical to the throughput ratio (221 / 87.2 ≈
2.5x). This is the clean, isolated proof that Part 6's mechanism is real
and it's expensive.

But look at the other two numbers. Padding simpleMultiThreadStoragePerThread
made it slightly slower — 1.247s versus 1.177s. Same claimed fix, same
underlying cache-line issue, and it did nothing. Worse than nothing. That's
not a contradiction — it's the most useful result in this entire article,
and it's worth understanding exactly why before reaching for alignas(64)
on everything.

Why the exact same fix helped in one case and did nothing in the other

The difference between the two benchmarks isn't the cache-line layout —
both genuinely have the false-sharing setup Part 6 describes. The
difference is how much work sits between one write and the next.

prac::FalseSharing's loop body is almost nothing: one fetch_add, loop
back, do it again. The cost of the cache-line ping-pong — tens of cycles
for a BusRdX round trip (Part 6) — is comparable to or larger than the
cost of the actual work being protected. There's nothing to hide the
contention behind, so it dominates the total time. That's exactly why the
throughput ratio (2.5x) lines up so cleanly with the time ratio: almost
every cycle in SharedLine's run is either doing the increment or waiting
on ownership of the line.

complexFunction's loop body is sin(cos(angle)) — real floating-point
transcendental math, on the order of dozens to hundreds of cycles per
iteration, every single time through the loop. Against that cost, the
occasional cache-line invalidation is comparatively small — it's still
happening, but it's a much smaller fraction of a much bigger number. The
expensive math is already keeping each core busy long enough that the
ownership ping-pong has less opportunity to actually stall anything; by the
time a core wants to write its counter again, there's a reasonable chance
the line has settled back to it, or the wait overlaps with work that would
have happened anyway.

flowchart LR
    A["Cheap loop body<br/>(fetch_add only)"] --> A2["Cache-line contention ≈<br/>or > cost of the work itself"]
    A2 --> A3["Contention dominates —<br/>2.5x slowdown, clearly visible"]

    B["Expensive loop body<br/>(sin/cos math)"] --> B2["Cache-line contention <<<br/>cost of the work itself"]
    B2 --> B3["Contention is noise —<br/>no measurable effect"]

    style A3 fill:#FAECE7,stroke:#993C1D,color:#4A1B0C
    style B3 fill:#E1F5EE,stroke:#0F6E56,color:#04342C

And the ~6% regression from padding AlignedCount isn't random noise to
wave away either — it has a real explanation. alignas(64) isn't free:
each AlignedCount now occupies a full 64-byte line instead of packing 16
to a line, which means more total memory touched, more lines pulled into
cache, and worse memory density overall. When the false-sharing cost you're
paying without padding is already smaller than the math dominating the
loop, padding adds real overhead to fix a problem that wasn't actually
costing you anything measurable — a pure loss.

The rule this leaves you with: false sharing's cost isn't fixed — it's
relative to how much work happens between writes. Cheap, frequent writes to
a contended line: the effect can dominate everything, like the 2.5x here.
Expensive work per write: the effect can vanish into the noise, or even
reverse once padding's own memory cost is added. There's no substitute for
benchmarking your actual access pattern before deciding alignas(64) is
worth paying for.

Why this is the strangest bug in the series so far

Every bug this series has covered until now had a correctness signal you
could point at: Part 1's race gave a wrong number. Part 2's compiler hazard
gave a hang or a skipped wait. This one gives neither. SharedLine and
PaddedLine both compute the exact right answer, every time, verified by
the if check in runTest itself. Nothing is wrong with the program's
output. The only symptom is time — the same correct answer, taking longer,
for a reason that appears nowhere in the C++ source.

This is exactly why Part 6 spent as long as it did on MESI and bus
transactions before this article ever showed a line of benchmark code:
without that vocabulary, "two threads with separate variables run slower
together than apart" looks like nonsense. With it, it's a direct, mechanical
consequence of one fact — cache lines are the real unit of ownership, not
your variables — playing out exactly as predicted.

Takeaways

Two threads can be completely race-free by every rule in Parts 1–2 and still contend, if their separate variables happen to share a cache line.
std::vector<int>'s contiguous layout is exactly the shape that causes this — Part 1's "fix" (per-thread storage) solved the correctness problem and unknowingly reintroduced a performance problem from a different layer.
alignas(64), with enough padding to actually fill the line, is the fix: force each hot variable onto its own cache line so writes never have to fight another core for ownership.
This bug produces no wrong answers, no crashes, no hangs — the only symptom is unexplained slowness, which makes it far easier to miss than anything else covered so far in this series.
The effect requires genuine multi-core contention to observe — it won't show up on a single-core machine or if the OS happens to schedule both threads on the same core.

Multi-Core Cache Coherency: The Invisible Lock You Never Wrote

Ajinkya Singh — Sun, 30 Aug 2026 18:53:30 +0000

Recap — and the question this article finally answers

Parts 3 through 5 built up a single-core picture: bytes and addresses, why
caches exist, and cache lines as fixed 64-byte aligned chunks. This article
introduces the piece that was missing — what happens when more than one
core wants the same memory at the same time — and it's the direct,
mechanical answer to the question this series has been sitting on since
Part 1: why did the mutex-protected version cost 13.2 seconds, over 4x
slower than not threading at all?

The new problem: private caches, shared memory

Registers, L1, and L2 are private per core. Only L3 and RAM are shared
across the whole chip (Part 4). That means the same 64-byte cache line can
end up copied into more than one core's private cache at the same time.

If nothing handled this, one core could write a value and another core
could read the same location moments later and get something already out
of date — not because of a bug in your code, purely because of where the
data happened to be cached. Hardware solves this fully automatically, with
no code from you required, via a coherency protocol.

The four states every cached line is tagged with (MESI)

The most common protocol family is called MESI, named after the four
states a cached line can be in on any given core:

Modified (M) — only this core has it, and it's dirty (differs from memory). Free to read or write. Nobody else has a copy.
Exclusive (E) — only this core has it, and it's clean (matches memory). Free to read; writing it silently flips it to Modified with zero bus traffic, since nobody else needs telling.
Shared (S) — this core and at least one other core have clean copies. Free to read. Cannot write without first evicting everyone else's copy — this is the expensive transition.
Invalid (I) — this core doesn't have a usable copy. The next access is a miss.

Every private cache runs this state machine independently, watching every
other core's traffic to know when to react. You don't configure it or see
it — it's just always running.

What actually crosses the wires

The state diagram shows the outcome. What physically happens on the
interconnect between cores is a small set of message types — every cache
"snoops" (watches) this traffic to see if it needs to react:

Message	Sent when...	Effect on other cores
BusRd (Read)	"I want to read a line I don't have"	holders downgrade to Shared; a Modified holder flushes first
BusRdX (Read-For-Ownership)	"I want to write a line I don't have"	every other copy anywhere is invalidated
BusUpgr (Upgrade)	"I already have it Shared, now I want to write"	every other copy invalidated — no data re-fetch needed, just permission
Flush	"I had it Modified and someone else needs it"	dirty data written back before giving up ownership

Here's the exact sequence two cores fighting over one line generate, every
time ownership swings back and forth:

Every BusRdX round trip costs on the order of the L3/interconnect
latency — tens of cycles — even though the actual arithmetic underneath is
a single cycle. This ping-pong is what's actually being measured whenever
two cores contend for the same line — not the math.

A cost ladder — not every core-to-core interaction is equal

Two cores sharing a line can meet in several different ways, and they are
not all the same price. Cheapest to most expensive:

Scenario	What happens	Relative cost
Local hit — already M/E on this core, no contention	Write happens purely in L1, zero bus traffic	Free
Cold read/write — nobody else has the line	Fetch from L3/DRAM, no one to invalidate	Cheap
Read, another core has it clean	Cache-to-cache copy, no invalidation	Moderate
Write, another core has it Shared (upgrade)	Permission-only request, no re-fetch	Moderate
Write, another core has it Exclusive/clean	Cache-to-cache transfer, invalidate, no flush	Moderate
Read, another core has it dirty	Full flush + cache-to-cache transfer	Expensive
Write, another core has it dirty	Full flush + invalidate + cache-to-cache transfer	Most expensive

The bottom row — write, while the other core is sitting on a dirty
(Modified) copy — is the one that matters most for this series:

Once two threads are both actively writing the same line, every single
write from either side lands in exactly this scenario — because whichever
side wrote last leaves the line Modified, which is precisely the expensive
case for whoever writes next. This back-and-forth, repeating on every write,
is the physical mechanism behind Part 1's 13.2 seconds: 400 million
lock/write cycles, each one paying this full flush-and-invalidate round
trip whenever the line had bounced to the other core since the last write.

The key insight: invalidation happens per line, not per variable

The coherency protocol doesn't know or care that a line contains "your
variable A" and "my unrelated variable B." It only knows this line was
written to — so it invalidates the whole line for every other core
holding it, even if those cores only ever touch a completely different
variable that just happens to live in the same 64 bytes.

Two threads can have zero shared variables in their own program logic —
no mutex, no atomic, nothing that looks like synchronization in the code at
all — and still slow each other down, purely because the hardware treats a
write to any part of a line as a reason to invalidate the entire line. This
isn't hypothetical. It's exactly what's waiting in the next article.

Wait — doesn't this mean coherency already prevents races?

Worth closing a gap explicitly, because it's a natural question once you've
seen how much machinery MESI actually runs: if hardware keeps every core's
cache honest automatically, why did Part 1's cnt += x still race?

Because coherency is a per-transaction guarantee, and cnt += x is not
one transaction — it's three (Part 1: read, modify, write). MESI correctly
guarantees that each individual read or write sees a consistent value.
It says nothing about a sequence of them. Nothing in MESI stops Core B
from sneaking in its own read between Core A's read and Core A's write —
there's no rule saying "once a core reads a line intending to write it
back, nobody else may touch that line until it does." That gap is exactly
where a race condition lives.

Every individual bus transaction above was perfectly coherent — the line
moved Shared → Modified → Invalid → Modified exactly as MESI dictates.
Coherency did its job at every single step. The bug isn't in the protocol;
it's in the gap between load and store, a gap coherency was never designed
to close. This is exactly what the lock prefix fixes and coherency alone
doesn't: it extends exclusive ownership across the whole read-modify-write,
not just one step of it, so no other core's read can slip into that gap.

Glossary added this part

Term	Meaning
Cache coherency	The guarantee that all cores see a consistent, up-to-date view of memory, even with private cached copies
MESI	Modified / Exclusive / Shared / Invalid — the four states a cached line can be in
BusRd / BusRdX / BusUpgr	The actual messages cores send to read, write, or upgrade permission on a line
Snoop	A cache watching bus traffic to see if it needs to react (invalidate, flush)
`lock` prefix	An x86 instruction prefix forcing a read-modify-write to hold exclusive line ownership for its entire duration

Takeaways

Cache coherency is real, automatic, and enforced entirely in hardware via a state machine (commonly MESI) and a small set of bus messages — no code of yours configures or sees it directly.
Not all core-to-core interactions cost the same. Writing while another core holds a dirty copy is the most expensive case there is — a full flush, invalidate, and transfer, every time — and it's exactly the case two threads land in on every write once they're actively contending for one line.
Coherency and atomicity are different guarantees. Coherency keeps a single access honest; atomicity (the lock prefix) keeps a sequence of accesses honest. cnt += x needs both, and plain non-atomic code only ever gets the first one — which is exactly why Part 1's race happened despite coherency running the whole time.
Invalidation happens per cache line, not per variable — two threads with zero shared variables in their own logic can still contend, purely from memory layout.

The 64-Byte Truck: Cache Lines Explained

Ajinkya Singh — Fri, 28 Aug 2026 19:23:55 +0000

Recap

Part 4 established why caching happens at all — spatial locality: if you
access one address, you're likely to want its neighbors soon, so hardware
fetches more than exactly what you asked for. This article answers the
obvious follow-up: how much more, exactly, and why that specific amount?

The answer is a fixed-size chunk called a cache line, and on most modern
CPUs that size is 64 bytes — always, no matter what you actually asked
for. This one fact is what the rest of this hardware track, and eventually
false sharing, is built entirely on top of.

What actually happens when you read one `int`

Say your code does int x = data[3]; and that triggers a cache miss — the
data isn't in cache yet, so it has to come from RAM. The natural mental
picture is the hardware reaching into RAM and pulling out exactly the 4
bytes that make up that one int. That is not what happens.

Instead:

The entire 64-byte block gets pulled in and cached as one unit, and your
int gets read out of that block afterward. You paid for 64 bytes whether
you needed them or not.

Why 64 bytes specifically — two independent reasons

Reason 1: the physical width of the connection

The hardware path between RAM and the CPU (and between cache levels) is
physically built to move a fixed number of bytes per transfer — its "width."
On most modern CPUs that width is 64 bytes. The critical part:
transferring 4 bytes over this path takes the same amount of time as
transferring all 64, because the limiting factor isn't the amount of data,
it's the fixed overhead of doing a transfer at all. Think of it like a
delivery truck that costs the same to dispatch whether it's carrying one box
or a full load — you fill the truck because the trip itself is the expensive
part, not the cargo.

Reason 2: spatial locality says it's worth it

Since nearby addresses are likely to be accessed soon anyway (Part 4), fetching
the surrounding 60 bytes "for free" — same transfer cost as fetching just 4 —
is a good bet almost all the time. Iterate over an array, and this pays off
constantly: fetching element 0 also fetches elements 1 through 15 (for 4-byte
ints), so the next several loop iterations are already cache hits before you
even ask.

Cache lines are aligned, not centered on what you asked for

A subtlety worth being precise about: the 64-byte chunk fetched isn't "the 64
bytes starting at the address you asked for." It's the 64-byte block that
address falls inside, where blocks are laid out in fixed segments starting
from address 0 (0-63, 64-127, 128-191, and so on). That's what aligned
means here.

So asking for the int at address 1004 doesn't fetch bytes 1004-1067 — it
fetches whichever fixed 64-byte segment contains address 1004 (addresses
960-1023, in this example). The requested bytes might sit anywhere inside
that segment — start, middle, or end.

This is the single most important detail in this article. Because lines
are fixed, aligned segments — not "64 bytes around whatever you asked for" —
any two variables that happen to land inside the same aligned 64-byte segment
get fetched and cached together as one unit, entirely regardless of whether
your program treats them as related at all. The hardware has no concept of
"these are two different variables." It only knows about lines.

How much fits in one line

64 bytes holds, for example:

16 ints (4 bytes each)
8 doubles (8 bytes each)
A handful of small struct fields, packed together

So in practice, several "neighboring" variables in a real program — array
elements, adjacent struct fields, even separate global or static variables
that happen to land near each other by the compiler's own layout choices —
very commonly share a single cache line without anyone ever writing code
that groups them together on purpose.

Glossary added this part

Term	Meaning
Cache line / cache block	The fixed-size chunk (typically 64 bytes) transferred and cached as one unit, every time
Bus width	The fixed amount of data a hardware connection moves per transfer, regardless of how much was actually requested
Aligned	Cache lines sit at fixed positions from address 0 (0-63, 64-127, ...), not centered on whatever address you asked for

Takeaways

A cache miss never fetches just what you asked for — it fetches the entire aligned 64-byte segment containing that address, every time.
64 bytes is fixed by hardware bus width (fetching 4 bytes costs the same as fetching 64, so hardware always fetches the full amount) and justified by spatial locality (the extra bytes are usually useful anyway).
Cache lines are aligned to fixed 64-byte boundaries starting from address 0 — not centered on your specific address — which is why unrelated variables can end up sharing a line purely by accident of memory layout.
The hardware has no concept of "your variables." It only tracks lines. Two completely unrelated ints can be neighbors in the same 64-byte block with neither programmer ever knowing.

The Hardware Memory Hierarchy: From L1 Caches to Main RAM

Ajinkya Singh — Sun, 23 Aug 2026 20:08:42 +0000

Recap

Part 3 left one number sitting there deliberately: a register access takes
roughly 1 cycle; a trip to main RAM takes roughly 200 cycles. This
article picks that gap up and asks the obvious next question — if RAM is
really 200x slower than a register, how does any real program run at a
reasonable speed at all? The answer is caches, and understanding why they
work is what makes the rest of this hardware track (and eventually, false
sharing) make sense instead of feeling like memorized trivia.

The problem caches solve

A CPU core executes an instruction in roughly 1 cycle. A single access to
main RAM takes roughly 200 cycles. If every memory access actually went all
the way to RAM, the CPU would spend the overwhelming majority of its time
just waiting — no matter how fast its execution units are.

Caches aren't an optimization bolted on as an afterthought. They're
load-bearing for a modern CPU to be useful at all.

The hierarchy, level by level

Level	Typical latency	Typical size	Scope
Registers	~1 cycle	Tens of bytes	Per core
L1 cache	~4 cycles	32-64KB	Per core (split: instructions / data)
L2 cache	~12 cycles	256KB-1MB	Per core
L3 cache	~40 cycles	A few MB to tens of MB	Shared across all cores on the chip
Main RAM	~200 cycles	GBs	Shared, off-chip

The pattern going down every step: size goes up roughly an order of
magnitude, latency goes up roughly 3-5x, and — this is the detail worth
underlining — what's visible to that level shrinks from "this core only"
(registers, L1, L2) to "every core on the chip" (L3, RAM). That shift, from
private per-core storage to shared storage, is the seed of the
entire cache-coherency problem two articles from now, and it's the reason a
mutex or an atomic operation touching L1-local data behaves completely
differently from one touching something every core has to agree on.

(Numbers vary by CPU generation and vendor — treat the table as
order-of-magnitude intuition, not a spec sheet for any specific chip.)

Why a pyramid, not one big fast memory

Why not just make all of RAM as fast as L1? Two physical reasons, not
engineering laziness:

Speed and size trade off directly in circuit design. The circuitry that makes L1 fast — more transistors per bit, shorter wire distances to the core — is also what makes it expensive and physically large per byte stored. Multi-gigabyte capacity at L1 speed isn't a software problem to solve; it's not economically or physically feasible with current technology.
Physical distance costs real time. Even at near-light-speed signal propagation, a memory bank farther from the core takes measurably longer to respond. L1 is built into the core. RAM sits on separate chips, connected over a bus. That distance alone adds latency, independent of the memory technology used.

So instead of one tier, you get a pyramid: a tiny amount of very fast storage
right next to the execution units, progressively larger and slower tiers
moving outward, with RAM as the big slow tier at the bottom.

How the hierarchy is actually used

When the CPU needs an address, it doesn't go straight to RAM — it checks
levels in order, fastest first:

A successful lookup at any level is a hit. Failing to find it and having
to check the next level down is a miss. Every miss that eventually
resolves — even all the way from RAM — copies the data upward into the
faster levels it passed through, on the expectation you'll want it again
soon.

Why caching works at all: locality

Caching only pays off because real programs exhibit locality —
predictable patterns in what memory they touch.

Temporal locality — if you accessed an address recently, you're likely to access it again soon. (A loop counter, read and written every iteration.)
Spatial locality — if you accessed an address, you're likely to soon access addresses near it. (Iterating an array — element 5 is immediately followed by element 6.)

Temporal locality is why keeping recently-used data around instead of
evicting it immediately is worth doing. Spatial locality is the direct
motivation for the next article: instead of fetching exactly the one
byte/int requested, hardware fetches a whole contiguous block around it —
because spatial locality says you'll probably want the neighbors too. That
block is called a cache line, and it's the single fact the rest of this
hardware track — and eventually false sharing — is built on.

Glossary added this part

Term	Meaning
Cache hit	Requested data found at this level — fast path
Cache miss	Requested data not found here — check the next, slower level
Latency	Time between requesting data and having it available
Temporal locality	Tendency to re-access the same address soon
Spatial locality	Tendency to access nearby addresses soon after one

Takeaways

Without caches, a CPU capable of 1-cycle instructions would spend nearly all its time waiting on 200-cycle RAM accesses. Caches aren't a bonus, they're required for a modern CPU to function at a usable speed.
The hierarchy is a pyramid, not a single fast tier, for two physical reasons: speed/size trade off in circuit design, and physical distance costs real time regardless of technology.
Registers, L1, and L2 are private per core. L3 and RAM are shared across every core on the chip — that shift from private to shared is exactly where multi-core problems start, two articles from now.
Caching only works because of locality — temporal (reuse soon) and spatial (neighbors soon) — and spatial locality specifically is why hardware always fetches more than you asked for.

How CPUs and Memory Actually Work: Bytes, Addresses, and Latency

Ajinkya Singh — Sat, 22 Aug 2026 21:04:53 +0000

Recap — the question this series has been avoiding

Part 1 left a number sitting there unresolved: the mutex-protected version of
complexFunction took 13.2 seconds — over 4x slower than running the
same work single-threaded with no threading at all. Part 2 showed a plain
bool racing across threads and getting hung or skipped entirely
depending on the optimization level. Both articles explained the C++-level
mechanics — read-modify-write, register caching, the compiler's license to
assume no data race exists — but neither one actually answered why a lock
costs that much time or why one core doesn't just immediately see another
core's write.

Those answers aren't in the C++ standard. They're in the hardware underneath
it. This article is where the series stops treating "memory" as a black box
and starts building the vocabulary to actually answer both open questions —
starting from the most basic thing there is: what a variable even is, once
you strip away the syntax.

A computer, stripped down to two things talking to each other

Underneath everything else, a running program is really just two components
in constant conversation:

The CPU — does the thinking. It can only perform very small, very simple operations: add two numbers, compare two numbers, move a number from one place to another. Everything complex your code does is millions of these tiny steps happening extremely fast, one after another.
Memory (RAM) — holds data while the program runs. The CPU itself can only hold a tiny amount of data internally at any given moment (registers, covered below), so anything larger — your variables, arrays, objects — lives out in memory, and the CPU has to go fetch it when it's needed and send it back when it's done.

That fetch-and-send-back relationship is the single most important thing to
carry forward from this article. Almost everything about performance and
multithreading correctness in this series comes back to "how fast can the
CPU get data out of memory, and what happens when more than one core wants
the same piece of memory at the same time."

Cycles — the CPU's metronome

A CPU doesn't work continuously; it advances on a beat. Each tick of that
beat is a clock cycle. On every tick, the CPU can move its work forward
by roughly one small step (a simplification, but the right mental model for
now).

Clock speed — the number on a spec sheet, like "3.5 GHz" — is just how many
of these ticks happen per second. 3.5 GHz means 3.5 billion ticks a second.
So when a later article says "this operation takes 200 cycles," that's not
an abstract unit — it's literally "the CPU sat through 200 of these ticks
before it could continue." At billions of ticks a second that's still a
tiny slice of real time, but it's 200 times longer than something that only
takes 1 tick, and that ratio is what actually matters, not the absolute
time.

Keep this number in your back pocket: reading from a CPU register takes
roughly 1 cycle. Reading from main RAM takes roughly 200 cycles.
That gap — the CPU sitting idle for 200 ticks of its own clock, doing
nothing but waiting for data — is the entire reason caches exist, and caches
are where this hardware track goes next.

Bytes and addresses — a variable is just a name for a location

Memory isn't one undivided blob. It's organized as a long row of small,
individually numbered slots:

A byte is the smallest individually addressable unit of memory — 8 bits. Most things you care about (an int, a char) are a small number of bytes glued together.
An address is the position number of a byte in memory — a house number on a very long street. Address 0 is the first byte, address 1 is the next, and so on.

Here's the reframe worth sitting with: int x = 42; is not, underneath
everything, a special protected thing called x. It's a label your compiler
uses for "start at this address, and the value there is 4 bytes long."
Reading x at runtime means the CPU is told an address and goes and fetches
whatever bytes are stored starting there.

This is the seed of everything coming later in the hardware track: two
completely unrelated variables in your program can end up sitting at
neighboring addresses, purely by how the compiler laid them out — with
consequences neither you nor the compiler intended. That's the exact setup
for false sharing, several articles from now.

Fast vs. slow — the ordering that everything else hangs on

You don't need exact nanosecond numbers yet — just the relative ordering,
because that ordering is the skeleton the rest of the hardware track hangs
real numbers on:

Each step to the right is slower to access but holds more data. That
trade-off — and exactly why the hierarchy is shaped like a pyramid instead of
one giant fast memory — is Part 5.

Cores — why "more than one CPU" changes everything

Modern CPUs usually contain more than one independent execution unit, each
able to run instructions on its own — a core. A 4-core CPU can run four
genuinely simultaneous streams of work, not just fast-switching between one
stream at a time. This is what makes Part 1's speedup real: each thread
lands on its own core, actually running in parallel.

But multiple cores introduce a genuinely new problem that doesn't exist with
a single core: each core typically has its own small private cache. So what
happens when two cores both want to read or write the same piece of memory
at the same time? That question — not anything about instructions or
compilers — is the real, hardware-level reason Part 1's mutex cost 13.2
seconds, and it's exactly what the next few parts of this series are for.

Takeaways

A CPU and RAM are, underneath everything, just two things constantly fetching from and sending data to each other — and that relationship is where every later performance question in this series comes from.
1 cycle for a register, ~200 cycles for RAM isn't a rounding error — it's the CPU sitting idle 200x longer, and it's the entire reason caches exist.
A variable is not a protected, special thing — it's just a name for "some bytes at some address," which means two unrelated variables can end up neighbors in memory without either programmer intending it.
Multiple cores make CPU-bound work genuinely parallel (Part 1's real speedup) but introduce a brand-new question single-core programs never had to answer: what happens when two cores want the same memory at once?

C++ Multithreading: Why -O2 Deletes Your Loops and How std::atomic Fixes It

Ajinkya Singh — Mon, 17 Aug 2026 21:22:33 +0000

Recap

Part 1 raced an int and got a wrong number — 66666666 instead of
266666664, a lost-update bug you could point at and explain with three
lines of assembly. Wrong, but at least it looked like arithmetic gone
wrong. This article races something simpler — a bool — and the result
isn't a wrong number at all. It's the compiler quietly throwing your loop
away.

The setup: a flag, and a thread that waits on it

The pattern: one thread does some slow work and then flips a flag. Another
thread just spins, checking the flag, waiting for it to become true.

struct nonAtomic { bool ready = false; };
struct atomic    { std::atomic<bool> ready{ false }; };

template<typename T>
void runTest(const char* label) {
    T data;
    std::thread t1([&data]() {
        std::this_thread::sleep_for(std::chrono::milliseconds(7000));
        data.ready = true;
    });
    while (!data.ready) { }

    std::cout << label << " - Finished waiting for ready flag.\n";

    t1.join();
}

t1 sleeps 7 seconds, then sets ready = true. The main thread sits in
while (!data.ready) { }, doing nothing but re-checking, until it sees
that flip. Two versions of the same test — one with a plain bool, one
with std::atomic<bool> — run back to back:

void run() {
    std::thread t1(&CompilerHazard::runTest<nonAtomic>, this, "nonAtomic");
    std::thread t2(&CompilerHazard::runTest<atomic>, this, "atomic");

    t1.join();
    t2.join();
}

Nothing here looks dangerous. There's no +=, no lost update possible — a
bool can only be true or false, there's no "partial" value to race
over the way there was with cnt. Reasonable to assume this one's fine.

The problem: the compiler doesn't just get this wrong — it gets it wrong differently at every optimization level

Compile and run this exact code at three optimization levels and you get
three different stories:

At -O0 (no optimization): both versions work. Each one waits roughly
7 seconds, then prints its "finished waiting" line. Boring, correct, exactly
what you'd expect from reading the source.

At -O1: the atomic version still works correctly. The nonAtomic
version hangs forever. Not "takes longer" — never returns. t1 finishes
its sleep, sets ready = true, and the main thread's loop never notices.

At -O2: the atomic version is still correct, still ~7 seconds. The
nonAtomic version does something even stranger than hanging — it finishes
instantly, without ever actually waiting for t1's 7-second sleep at
all.

Same source code. Same hardware. Only the optimization flag changed, and
the plain-bool version gave three different, non-equivalent results
across three builds — one correct, one an infinite hang, one a nonsensical
instant "finish" that skipped the wait entirely.

Why: the compiler is allowed to assume this bug doesn't exist

while (!data.ready) { } re-checks ready on every loop iteration, hoping
to notice when another thread changes it. That's the intent.

But ready is a plain bool. One thread writes it, another thread reads
it, and nothing synchronizes the two. That's the exact definition of a data
race from Part 1 — and in C++, a data race is undefined behavior for the
entire program, not just that one variable.

Here's the part that trips people up: "undefined behavior" doesn't mean
"the compiler gives up." It means the compiler is legally allowed to
assume this situation never happens in a correct program. As far as the
optimizer can see, nothing in this function ever modifies ready once the
loop starts — so it treats ready as if it were constant for the rest of
the loop's life.

That one assumption plays out differently depending on how hard the
compiler is optimizing:

At -O1: the compiler reads ready once, stores it in a CPU register,
and just keeps re-checking that register forever — it never goes back out
to memory to look for t1's update. while (!data.ready) { } quietly
becomes while (true) { }. The check isn't wrong; it's just checking a
value that can never change.

At -O2: the compiler goes a step further. If no other thread is
allowed to touch ready, then the branch where this loop waits forever
for an external change is impossible — and impossible code is free to be
deleted. So it deletes the wait entirely and falls straight through, as if
the condition were already true.

Neither of these is a compiler bug. Both are legal optimizations, built on
a promise your code made by not synchronizing ready — and then broke.

`std::atomic` actually does two different jobs — don't blur them together

Here's where it's easy to get turned around: Part 1 fixed a race on cnt++.
This article is about a race on ready = true. Both get called "a data
race," both get fixed by std::atomic, and it's tempting to assume
std::atomic is one trick doing the same thing both times. It isn't. It's
two different guarantees, and each bug only needed one of them.

Job 1 — Atomicity: make a multi-step operation indivisible.
This is Part 1's problem. cnt += x is three instructions — read, modify,
write — and another thread can slip in between them. std::atomic<int>
fixes this by compiling the whole read-modify-write into one
hardware-locked instruction that cannot be interrupted:

; plain int:  cnt += x  -->  three separate, interruptible steps
mov  eax, [cnt]
add  eax, edx
mov  [cnt], eax

; std::atomic<int>:  cnt.fetch_add(x)  -->  one indivisible step
lock xadd [cnt], eax

The lock prefix tells the CPU: no other core may touch this memory
location until this single instruction finishes. There's no longer a
"between the read and the write" for another thread to land in, because
there's no longer a between at all.

Job 2 — Visibility: make a completed write reachable by other cores, in
order.
This is this article's problem. ready = true is already one
instruction — there's nothing to make indivisible, it's not being
interrupted mid-write. The bug isn't timing, it's that the compiler cached
a stale read, and separately, a real write can sit in a core's private
cache without a guaranteed moment of becoming visible elsewhere.
std::atomic<bool> fixes this by stopping the compiler from caching the
read, and by inserting the memory-ordering guarantees that make the write's
completion visible to other cores in a well-defined way.

Same keyword, two different repairs, applied depending on what's actually
wrong with the operation you wrapped in std::atomic<>. A single store
never needed Job 1 — it was already indivisible. A multi-step
read-modify-write always needs Job 1, and usually gets Job 2 along with it
for free, since std::atomic provides both by default.

Back to `ready` — this article only ever needed Job 2

Since ready = true was already one instruction, there was nothing here
for atomicity to fix. The actual gap was purely visibility:

Each core has its own cache. A write can sit there for a while before it's
visible to any other core. Without a synchronization point, there's no
promise about when — or even in what order — one core's writes show up to
another core's reads. std::atomic<bool> closes that gap: not by making a
single store "more atomic" (it already was one instruction), but by forcing
that store to become visible to other cores in a defined, guaranteed way,
and by stopping the compiler from short-circuiting the check with a cached
register value.

The full "why" behind cross-core visibility — cache lines, per-core caches,
the coherency protocol that keeps them honest — is what the Hardware Track
later in this series is for. For now, the one thing worth keeping straight:
"atomic" doesn't mean one fix for one kind of bug. It means "pick the
guarantee your specific operation is missing," and std::atomic<T> gives
you both, whichever one your code actually needed.

The fix: `std::atomic<bool>`

struct atomic {
    std::atomic<bool> ready{ false };
};

Same loop, same while (!data.ready) { }, only the type changes — and it
works at every optimization level, every time. For this specific bug, that's
entirely Job 2 doing the work: no more caching ready in a register across
iterations, and a guaranteed, ordered handoff of the write from one core to
the other.

std::atomic isn't "the fix for wrong numbers" or "the fix for hangs"
specifically — it's a toolbox with two tools in it, and which one your bug
needs depends on whether the operation you're protecting is multiple steps
(atomicity) or a single step whose visibility isn't guaranteed (ordering).
Wrong numbers, infinite hangs, and skipped checks are just different
costumes the same root cause — an unsynchronized shared variable — can wear,
depending on which tool was actually missing.

Why this matters more than the Part 1 bug did

Part 1's bug was loud, in a sense — the number was wrong, and if you had a
known-correct baseline to compare against (like we did), you'd catch it. This
bug is worse precisely because its behavior isn't even stable across
builds. Code that hangs at -O1 might look completely fine at -O0 in a
debug build during development, then hang in a release build in production.
Code that races at -O2 might not even wait for the condition it was
written to wait for, and silently proceed on data that isn't ready yet —
no crash, no hang, no obvious signal that anything went wrong at all.

"It worked when I tested it" was never proof of anything for a data race.
It's proof that this specific compiler, at this specific optimization
level, on this specific run, didn't happen to apply the optimization that
breaks you. Change any one of those three things and the same source code
can behave completely differently.

Takeaways

std::atomic does two distinct jobs, not one: atomicity (making a multi-step operation like cnt++ indivisible, via a locked instruction) and visibility (making a single-step write's completion reliably reach other cores, plus stopping the compiler from caching stale reads).
Part 1's cnt++ needed atomicity — it's three instructions with a gap to race in. This article's ready = true needed visibility — it's already one instruction, nothing to make more indivisible.
The exact failure mode (hang, instant skip, or correct-by-luck) can change across optimization levels for the same source file — there is no single "how it breaks," which is what makes this category of bug so hard to trust-by-testing.
Debug builds (-O0) hiding this class of bug entirely is a real and common trap — "no threading bugs in debug mode" tells you nothing about release mode.

Understanding C++ Threads: Data Races Explained Down to the Assembly

Ajinkya Singh — Sun, 16 Aug 2026 18:24:34 +0000

Notes & References

Sources this series builds on: Chili's (ChiliTomatoNoodle) multithreading
playlist, CMU 15-213 (Computer Systems: A Programmer's Perspective), and various CppCon talks.

Everything here is my own code, compiled and run — not just transcribed. These are my personal notes and references for future programming. LLMs were utilized for restructuring and sanity-checking the explanations.

The setup

One expensive, CPU-bound function. Every version in this article calls it —
only how many threads and how they share state changes.

void Thread::complexFunction(int& cnt) {
    const double PI_HALF = 1.5707963267948966;
    for (int i = 0; i < 100000000; ++i) {
        double angle = (double(i) / 100000000) * PI_HALF;
        cnt += (int)(std::sin(angle) * 2);
    }
}

Run it 4 times, back to back, on one thread:

void Thread::singleThread() {
    t.start();
    int cnt = 0;
    for (int i = 0; i < 4; i++) {
        complexFunction(cnt);
    }
    t.stop("Single Thread", true);
    std::cout << "cnt: " << cnt << "\n";
}

[Single Thread]: 3.04755 seconds
cnt: 266666664

That number, 266666664, is our source of truth for the rest of this
article. Every "correct" version below has to reproduce it.

The problem: the obvious multithreaded version is wrong

The natural next move: instead of calling complexFunction 4 times on one
thread, launch 4 threads and let them all run it at the same time, all
writing into the same cnt.

void Thread::simpleMultiThreadRaceCondition() {
    t.start();
    int cnt = 0;
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnt)));
    }
    for (auto& th : threads) {
        if (th.joinable()) {
            th.join();
        }
    }
    t.stop("Multi Thread Race Condition", true);
    std::cout << "cnt: " << cnt << "\n";
}

(Quick note on the mechanics here, since this is the first threaded code in
the series: std::ref(cnt) is required because std::thread copies its
arguments by default, and a plain int& parameter wouldn't survive that —
std::ref forces the real reference through. .join() in the loop after is
what blocks main until every worker is actually finished; skipping it and
letting threads go out of scope early is an instant crash, not a subtle bug.)

Run it:

[Multi Thread Race Condition]: 1.18949 seconds
cnt: 66666666

Faster, sure — that's expected, 4 cores instead of 1. But 66666666 is not
266666664. It's not close. It's suspiciously close to exactly
one quarter of the right answer — as if 3 out of every 4 threads'
work just vanished. The code compiled fine, ran fine, crashed nothing. It
just handed back the wrong number, silently.

Why: `cnt += x` is not one instruction

Here's the thing to unlearn: cnt += (int)(std::sin(angle) * 2); looks
like a single step in your source code. It is not one step for the CPU. At
-O0, it compiles to three separate instructions:

mov  eax, [cnt]      ; READ:   load cnt's current value into a register
add  eax, edx        ; MODIFY: add the new value to it
mov  [cnt], eax       ; WRITE:  store the result back into cnt

Read the value. Modify it. Write it back. Three distinct steps, three
distinct moments in time — and on a 4-core machine, all four threads are
doing this to the same address, at the same time, with no rule saying
one thread's three steps have to finish before another thread's three steps
begin.

Think of it like four people sharing one notebook with a running total
written in it, and no rule about taking turns:

Person A reads the notebook: "it says 100."
Before A writes anything back, Person B also reads it: "it also says 100."
A computes 100 + 3 = 103, and writes 103 into the notebook.
B, still working off the 100 it read a moment ago, computes 100 + 5 = 105, and writes 105 into the notebook — overwriting A's update completely.

Nobody gets an error. Nothing crashes. The notebook just silently ends up
wrong, and there's no record that A's update ever happened at all. This is
called a lost update, and it's what's known as a data race: two
threads touching the same memory, at least one of them writing, with no
coordination between them at all.

Do that millions of times a second, across four threads, for 100 million
iterations each, and you get a final number that isn't "slightly off" — it's
wrong in a way that happens to look patterned (close to 1/4 of the true
answer) purely because of how evenly the four threads' work happened to
collide on this run. Run it again and you'd likely get a different wrong
number. That's the part that should feel uncomfortable: the wrongness
itself isn't even consistent.

Solution attempt 1: lock the notebook — a mutex

If the problem is four threads reading and writing the same memory with no
coordination, the direct fix is to add coordination: force only one thread
at a time to be allowed inside the read-modify-write, no matter how many
threads are running.

void Thread::complexFunctionWithMutex(int& cnt) {
    const double PI_HALF = 1.5707963267948966;
    for (int i = 0; i < 100000000; ++i) {
            std::lock_guard<std::mutex> lock(mtx);
            double angle = (double(i) / 100000000) * PI_HALF;
            cnt += (int)(std::sin(angle) * 2);
    }
}

void Thread::simpleMultiThreadMutex() {
    t.start();
    int cnt = 0;
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunctionWithMutex, this, std::ref(cnt)));
    }
    for (auto& th : threads) {
        if (th.joinable()) {
            th.join();
        }
    }
    t.stop("Multi Thread Mutex", true);
    std::cout << "cnt: " << cnt << "\n";
}

std::lock_guard<std::mutex> lock(mtx); is the notebook's lock: whichever
thread grabs it first gets to finish its entire read-modify-write
uninterrupted, and every other thread has to wait its turn. No two threads
can be mid-update at the same time anymore — the exact gap that caused the
lost update is closed.

[Multi Thread Mutex]: 13.2063 seconds
cnt: 266666664

Correct — matches the single-threaded baseline exactly, every run. But look
at the time: 13.2 seconds, against a single-threaded baseline of 3.05
seconds. Using 4 cores made this over 4x slower than using one core and
no threading at all.

The reason is that we locked the mutex inside the loop, on every single
iteration — 400 million lock/unlock operations total, most of them spent
with three threads waiting idle for the fourth to finish its turn. We didn't
just add coordination — we added coordination so fine-grained that the four
threads are barely running in parallel at all; they're mostly taking turns,
plus paying the overhead of acquiring a lock 400 million times on top of that.

Correctness, fixed. Performance, worse than not threading in the first
place. That's a real trade-off, and it's hit constantly:
a mutex fixes the wrong answer problem, but a careless mutex can make your
"parallel" code slower than serial code.

Solution attempt 2: don't share the notebook at all

Here's the question worth sitting with: did these four threads ever actually
need to share cnt while they were working? Each thread's math is
completely independent of the other three — thread 2 never needs to know
what thread 1 computed mid-loop. The only reason we had a race at all is
that we chose to point all four threads at the same variable.

So — don't:

void Thread::simpleMultiThreadStoragePerThread() {
    t.start();
    int sum = 0;
    std::vector<int> cnts(4, 0);
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnts[i])));
    }

    for (auto& t : threads) {
        if (t.joinable()) {
            t.join();
        }
    }

    for (auto& cnt : cnts) {
        sum += cnt;
    }
    t.stop("Multi Thread Storage Per Thread", true);
    std::cout << "cnt: " << sum << "\n";
}

Give each thread its own slot (cnts[i]) instead of a shared one, let all
four run completely uninterrupted with no locking at all, and only add the
four totals together after every thread has joined — at which point no
thread is writing anymore, so there's nothing left to race.

[Multi Thread Storage Per Thread]: 1.12478 seconds
cnt: 266666664

Correct, and the fastest version by a clear margin — faster than the mutex
version by more than 10x, and even slightly faster than the naive (wrong)
race-condition version, because there's no locking overhead at all.

The full picture

Version	Time	`cnt`	Correct?
Single Thread	3.05s	266666664	✅ baseline
Race Condition (shared `cnt`, no lock)	1.19s	66666666	❌ lost updates
Mutex (shared `cnt`, locked every iteration)	13.2s	266666664	✅ but 4x slower than serial
Storage Per Thread (no sharing at all)	1.12s	266666664	✅ fastest overall

The lesson isn't "always use a mutex" or "always avoid shared state" — it's
that the fastest fix for a race is often to make the race impossible
rather than to police it. A mutex is the right tool when threads
genuinely need to coordinate over shared, mutable state. Here, they never
did — the sharing was the bug, not something that needed defending.

Takeaways

x += y is read-modify-write at the hardware level, not one atomic step — that gap between "one line of C++" and "three real instructions" is where every data race in this series starts.
A data race doesn't announce itself. It can produce a plausible-looking wrong number, and a different wrong number on the next run.
A mutex restores correctness by serializing access — but locking too finely (every iteration, in this case) can make multithreaded code slower than not threading at all.
If threads don't actually need to share mutable state, removing the sharing beats synchronizing it — no lock, no contention, no overhead.

Kubernetes Core Workloads: Everything You Need to Know

Ajinkya Singh — Fri, 03 Apr 2026 07:52:31 +0000

If you've made it past the basics of Kubernetes — you know what a cluster is, you've spun up a local environment with kind or minikube — the next wall you hit is understanding the workload objects. There are nine of them. They look similar in YAML. They all involve containers. But they each exist for a very different reason.

The Quick Reference: Which Workload Do You Need?

Before diving in, here's the decision tree you'll use every day:

Workload	Use When
Deployment	Running stateless apps, APIs, web servers, workers
StatefulSet	Databases, message brokers, anything needing stable identity
DaemonSet	Log agents, monitoring — one pod per node
Job	One-off batch tasks: migrations, reports, cleanup
CronJob	Scheduled recurring tasks
Pod (bare)	Almost never in production — use one of the above

Now let's understand why.

Pods: The Atomic Unit

Everything in Kubernetes ultimately runs as a Pod — the smallest deployable unit. You don't run containers directly; you run Pods that contain containers.

Think of a Pod as a tiny apartment that one or more containers share:

They share a single IP address
Containers within a Pod talk to each other via localhost
They can share volumes (disk storage)
They always land on the same node — never split

apiVersion: v1
kind: Pod
metadata:
  name: api-server-pod
spec:
  containers:
    - name: api
      image: mycompany/api:2.1.0
      resources:
        requests:
          memory: "256Mi"
          cpu: "250m"
        limits:
          memory: "512Mi"
          cpu: "1000m"
      livenessProbe:
        httpGet:
          path: /health
          port: 3000
        initialDelaySeconds: 15
        periodSeconds: 20
      readinessProbe:
        httpGet:
          path: /ready
          port: 3000
        initialDelaySeconds: 5
        periodSeconds: 10

The Most Important Thing About Pods

Pods are ephemeral. When a Pod dies, a new one is created with a completely new name and a completely new IP. Any code that hardcoded the old IP breaks. This is why Services exist (Phase 3) — but it's the mental model you need to carry through everything else.

Two Probes You Can't Skip

Liveness probe — "Is this container still alive?" If it fails, Kubernetes restarts the container. Use for crash detection and deadlocks.

Readiness probe — "Is this container ready for traffic?" If it fails, the pod is removed from the Service's endpoints — stops receiving traffic — but is not restarted. Use for startup warmup and temporary overload.

These two probes are doing different jobs. Never confuse them.

Pod Status Lifecycle

Pending → Running → Succeeded
              ↓
         Failed / CrashLoopBackOff / OOMKilled

When you see CrashLoopBackOff, run kubectl logs my-pod --previous — that's where your crash output lives.

Quick Debugging Playbook

# Pod stuck in Pending?
kubectl describe pod my-pod   # read the Events section
# "Insufficient CPU" → reduce requests or add nodes

# ImagePullBackOff?
kubectl describe pod my-pod   # wrong image name? typo? private registry?

# CrashLoopBackOff?
kubectl logs my-pod --previous   # read the crash

# OOMKilled?
kubectl describe pod my-pod   # see Last State → increase memory limit

# Running but not working?
kubectl exec -it my-pod -- /bin/bash   # investigate from inside

🧪 Practice — Pods

Lab 1: Your first pod

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: hello-nginx
  labels:
    app: hello
spec:
  containers:
    - name: nginx
      image: nginx:1.25
      ports:
        - containerPort: 80
      resources:
        requests:
          memory: "64Mi"
          cpu: "50m"
        limits:
          memory: "128Mi"
          cpu: "200m"
EOF

kubectl get pods --watch
kubectl describe pod hello-nginx    # read the Events section top to bottom
kubectl exec -it hello-nginx -- /bin/bash
# inside: curl localhost
kubectl port-forward pod/hello-nginx 8080:80 &
curl http://localhost:8080
kill %1

Lab 2: Multi-container pod — shared localhost

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: multi-pod
spec:
  containers:
    - name: app
      image: nginx:1.25
    - name: sidecar
      image: busybox
      command: ['sh', '-c', 'while true; do echo sidecar running; sleep 5; done']
EOF

kubectl logs multi-pod -c app
kubectl logs multi-pod -c sidecar
kubectl exec -it multi-pod -c sidecar -- /bin/sh
# inside: wget -q -O- localhost   ← hits the nginx container via shared IP

Lab 3: Trigger and observe CrashLoopBackOff

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: crasher
spec:
  containers:
    - name: app
      image: busybox
      command: ['sh', '-c', 'echo crashing; exit 1']
EOF

kubectl get pods --watch    # watch it go into CrashLoopBackOff
kubectl logs crasher --previous   # see the crash output

Cleanup

kubectl delete pod hello-nginx multi-pod crasher

Namespaces: Logical Isolation

A Namespace is a virtual cluster inside your real cluster. It's how you divide one Kubernetes cluster into isolated sections — by team, environment, or application.

Four namespaces exist by default:

default — where objects land if you don't specify
kube-system — Kubernetes system components (etcd, apiserver, coredns)
kube-public — publicly readable data (rarely used)
kube-node-lease — node heartbeat objects (internal, ignore)

What Namespaces Give You

Name reuse — you can have a webapp Deployment in both staging and production with no conflict.

RBAC scoping — give team A access only to their namespace.

Resource quotas — cap a namespace to 4 CPUs and 8Gi memory total.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    pods: "50"
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi

One Critical Misconception

Namespaces do not provide network isolation by default. A pod in dev can still reach a pod in prod if it knows the IP or DNS name. For actual network isolation, you need NetworkPolicies.

Cross-namespace DNS format: <service>.<namespace>.svc.cluster.local

🧪 Practice — Namespaces

Lab: Isolation, quotas, and the -n flag

# Create two environments
kubectl create namespace dev
kubectl create namespace prod

# Deploy the same app name in both — no conflict
kubectl create deployment webapp --image=nginx:1.25 -n dev
kubectl create deployment webapp --image=nginx:1.25 -n prod

kubectl get deployments -A    # see both side-by-side

# Apply a quota to prod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ResourceQuota
metadata:
  name: prod-quota
  namespace: prod
spec:
  hard:
    pods: "5"
    requests.cpu: "2"
    requests.memory: 4Gi
EOF

# Try to exceed it
kubectl scale deployment webapp --replicas=10 -n prod
kubectl get pods -n prod              # stops at quota limit
kubectl describe resourcequota prod-quota -n prod   # usage vs hard

# Set dev as your default namespace
kubectl config set-context --current --namespace=dev
kubectl get pods    # now implicitly reads from dev

# Switch back
kubectl config set-context --current --namespace=default

Cleanup

kubectl delete namespace dev prod

Labels, Selectors & Annotations

Labels are key-value pairs on any Kubernetes object. They're the connective tissue — how Services find Pods, how Deployments track their ReplicaSets, how you filter resources.

Annotations are also key-value pairs but for non-identifying metadata: descriptions, tool config, URLs. They cannot be used in selectors.

metadata:
  labels:
    app: api
    env: production
    version: "2.1.0"   # always quote values that look like numbers
  annotations:
    description: "Payments API server"
    prometheus.io/scrape: "true"

How a Service Uses Labels

apiVersion: v1
kind: Service
metadata:
  name: backend-service
spec:
  selector:
    app: api           # matches pods with app=api
    env: production    # AND env=production
  ports:
    - port: 80
      targetPort: 3000

The Service routes traffic to any pod with both labels. Remove a label from a pod and it's immediately removed from the Service's endpoints. Add it back and it rejoins. This is live — no restart required.

Set-Based Selectors

Beyond simple equality, you can use expressions:

selector:
  matchExpressions:
    - key: env
      operator: In
      values: [production, staging]
    - key: tier
      operator: NotIn
      values: [frontend]
    - key: app
      operator: Exists

One gotcha: always quote version numbers in YAML. version: 2.0 is parsed as a float, stored as "2" — breaking your selectors silently.

🧪 Practice — Labels & Selectors

Lab: Live label manipulation and Service routing

# Create three pods with different label combos
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: frontend-v1
  labels:
    app: frontend
    env: prod
    version: "1.0"
spec:
  containers:
    - name: nginx
      image: nginx:1.25
---
apiVersion: v1
kind: Pod
metadata:
  name: backend-v1
  labels:
    app: backend
    env: prod
    version: "1.0"
spec:
  containers:
    - name: nginx
      image: nginx:1.24
---
apiVersion: v1
kind: Pod
metadata:
  name: backend-staging
  labels:
    app: backend
    env: staging
    version: "2.0"
spec:
  containers:
    - name: nginx
      image: nginx:1.26
EOF

# Filter practice
kubectl get pods --show-labels
kubectl get pods -l app=backend
kubectl get pods -l app=backend,env=prod
kubectl get pods -l 'env in (prod,staging)'
kubectl get pods -l 'version notin (1.0)'

# Create a Service that selects app=backend + env=prod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: backend-svc
spec:
  selector:
    app: backend
    env: prod
  ports:
    - port: 80
      targetPort: 80
EOF

kubectl describe service backend-svc   # check Endpoints section

# Live label surgery — remove env from backend-v1
kubectl label pod backend-v1 env-
kubectl describe service backend-svc   # endpoint disappears immediately

# Add it back
kubectl label pod backend-v1 env=prod
kubectl describe service backend-svc   # endpoint returns

Cleanup

kubectl delete pod frontend-v1 backend-v1 backend-staging
kubectl delete service backend-svc

ReplicaSets: Self-Healing

A ReplicaSet ensures a specified number of identical Pods are always running.

spec.replicas: 3

Pod-2 crashes → ReplicaSet creates Pod-4 → back to 3
Rogue pod added → ReplicaSet deletes one → back to 3

The RS continuously counts pods matching its selector. Actual count ≠ desired → create or delete.

Why You Almost Never Create ReplicaSets Directly

ReplicaSets don't support rolling updates or rollbacks. That's exactly what Deployments add. In practice, you create a Deployment and it creates/manages a ReplicaSet for you.

You need to understand ReplicaSets because:

Deployments own them under the hood
Debugging often means inspecting the RS
It explains the self-healing behavior you see

The RS name format when created by a Deployment: <deployment-name>-<pod-template-hash>. That hash changes every time the pod template changes — which is exactly how rolling updates work.

🧪 Practice — ReplicaSets

Lab: Self-healing in action

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: demo-rs
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
EOF

kubectl get pods -l app=demo

# Open a watch in a second terminal: kubectl get pods --watch

# Kill one pod — RS heals immediately
kubectl delete pod $(kubectl get pods -l app=demo -o jsonpath='{.items[0].metadata.name}')
kubectl get pods -l app=demo    # new pod already created

# Scale up
kubectl scale rs demo-rs --replicas=5
kubectl get pods -l app=demo

# Try creating a stray pod with the same label
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: stray-pod
  labels:
    app: demo        # matches RS selector!
spec:
  containers:
    - name: nginx
      image: nginx:1.25
EOF

kubectl get pods -l app=demo    # RS sees 6, wants 5 → deletes one

Cleanup

kubectl delete rs demo-rs

Deployments: The Standard Way to Run Apps

A Deployment is what you use 90% of the time. It wraps a ReplicaSet and adds rolling updates, rollbacks, and version history.

You → Deployment → manages → ReplicaSets → manages → Pods
                      │
                      ├── RS v1 (old image)  → 0 pods
                      └── RS v2 (new image)  → 3 pods ✅

How a Rolling Update Actually Works

With maxSurge: 1, maxUnavailable: 0 and 3 replicas:

[v1][v1][v1]               # start

[v1][v1][v1][v2]           # spin up 1 new pod (surge)
[v1][v1][v2]               # v2 passes readiness → kill 1 v1

[v1][v1][v2][v2]           # spin up another v2
[v1][v2][v2]               # v2 passes → kill v1

[v1][v2][v2][v2]           # final new pod
[v2][v2][v2]               # all updated, zero downtime ✅

The readiness probe gates each step. Without it, Kubernetes can't tell when a new pod is actually ready.

Production Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0   # never go below desired count
  minReadySeconds: 10
  revisionHistoryLimit: 5
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: api
          image: mycompany/orders-api:3.0.1
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 5

The Commands You'll Use Daily

# Update image → triggers rolling update
kubectl set image deployment/my-app app=nginx:1.26

# Watch it happen
kubectl rollout status deployment/my-app

# Something went wrong?
kubectl rollout undo deployment/my-app

# Check history
kubectl rollout history deployment/my-app

# Restart all pods (rolling, with zero downtime)
kubectl rollout restart deployment/my-app

kubectl rollout undo is your emergency brake. Run it first, debug second.

🧪 Practice — Deployments

Lab: Full deployment lifecycle — deploy, update, break, rollback

# Deploy v1
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  annotations:
    kubernetes.io/change-cause: "initial deployment nginx 1.24"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
        - name: nginx
          image: nginx:1.24
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
EOF

kubectl get pods --watch    # Ctrl+C when all 3 Running
kubectl get rs              # see the RS created by the Deployment

# Rolling update to 1.25
kubectl set image deployment/webapp nginx=nginx:1.25
kubectl annotate deployment/webapp kubernetes.io/change-cause="upgrade to nginx 1.25" --overwrite
kubectl rollout status deployment/webapp
kubectl rollout history deployment/webapp   # see 2 revisions
kubectl get rs    # old RS now has 0 pods but still exists for rollback

# Simulate a bad update — nonexistent image
kubectl set image deployment/webapp nginx=nginx:NONEXISTENT
kubectl rollout status deployment/webapp    # hangs — new pods can't start
kubectl get pods    # old pods still serving! new pods in ErrImagePull

# Emergency rollback
kubectl rollout undo deployment/webapp
kubectl rollout status deployment/webapp    # watch recovery
kubectl get pods    # all healthy again

# Scale
kubectl scale deployment webapp --replicas=6
kubectl scale deployment webapp --replicas=2

Cleanup

kubectl delete deployment webapp

Imperative vs Declarative: The Mental Model

There are two ways to work with Kubernetes.

Imperative — tell Kubernetes what to do step by step:

kubectl create deployment web --image=nginx:1.25
kubectl scale deployment web --replicas=3
kubectl set image deployment/web nginx=nginx:1.26

Declarative — tell Kubernetes what you want, let it figure out how:

kubectl apply -f web.yaml
# edit the file
kubectl apply -f web.yaml   # kubernetes computes the diff

The key advantage of declarative: idempotency.

# Imperative — fails on second run
kubectl create deployment web --image=nginx
# Error: deployments.apps "web" already exists

# Declarative — safe to run forever
kubectl apply -f web.yaml
# deployment.apps/web created      (first run)
# deployment.apps/web unchanged    (no changes)
# deployment.apps/web configured   (file changed)

In production: always declarative. Store YAML in Git. CI/CD runs kubectl apply. This is GitOps — Git is the source of truth, the cluster always mirrors it.

Generate YAML Fast

kubectl create deployment web --image=nginx:1.25 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

--dry-run=client -o yaml generates the YAML locally without sending anything to the API server. Edit it, then apply. This is extremely useful in CKA/CKAD exams.

🧪 Practice — Imperative vs Declarative

Lab: Generate, edit, and apply

# Generate Deployment YAML without applying
kubectl create deployment web --image=nginx:1.25 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

cat deployment.yaml    # inspect what was generated

# Add resources block manually, then apply
kubectl apply -f deployment.yaml
kubectl apply -f deployment.yaml    # safe to run again — "unchanged"

# Edit the file — change replicas to 5
sed -i 's/replicas: 3/replicas: 5/' deployment.yaml
kubectl apply -f deployment.yaml    # "configured" — only diff applied
kubectl get deployment web

# Generate Pod and Service YAML for practice
kubectl run my-pod --image=busybox --dry-run=client -o yaml > pod.yaml
kubectl expose deployment web --port=80 --type=ClusterIP \
  --dry-run=client -o yaml > service.yaml

Cleanup

kubectl delete deployment web

DaemonSets: One Pod per Node

A DaemonSet ensures exactly one Pod runs on every node in the cluster. New node joins → pod auto-created. Node removed → pod garbage collected.

Node 1      Node 2      Node 3      Node 4 (new)
[log]       [log]       [log]       [log] ← auto-created

When to Use DaemonSets

Use Case	Examples
Log collection	Fluentd, Filebeat, Promtail
Monitoring agents	Prometheus node-exporter, Datadog agent
Network plugins	Calico, Cilium, kube-proxy
Security agents	Falco, Sysdig

kube-proxy itself is a DaemonSet. Verify: kubectl get ds -n kube-system

Running on Control-Plane Nodes

By default, DaemonSet pods won't run on control-plane nodes because of a taint: node-role.kubernetes.io/control-plane:NoSchedule. If you need them there, add a toleration:

spec:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

Update Strategies

RollingUpdate (default): automatically updates pods one node at a time
OnDelete: only updates pods when you manually delete them — useful for per-node maintenance windows

🧪 Practice — DaemonSets

Lab: Per-node coverage and system DaemonSets

# See existing DaemonSets in the cluster
kubectl get ds -n kube-system -o wide
# kube-proxy is a DaemonSet — one pod per node

# Create a simple node-monitoring DaemonSet
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-monitor
spec:
  selector:
    matchLabels:
      app: node-monitor
  template:
    metadata:
      labels:
        app: node-monitor
    spec:
      containers:
        - name: monitor
          image: busybox
          command:
            - sh
            - -c
            - while true; do echo "Node $(NODE_NAME) alive at $(date)"; sleep 10; done
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests:
              memory: "20Mi"
              cpu: "10m"
EOF

# Verify one pod per worker node
kubectl get pods -l app=node-monitor -o wide

# Check logs from one
POD=$(kubectl get pods -l app=node-monitor -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD

Cleanup

kubectl delete ds node-monitor

Jobs & CronJobs: Batch and Scheduled Work

Job — runs pods to completion. Once finished successfully, stops. For one-off batch tasks.

CronJob — runs a Job on a schedule (cron syntax). For recurring tasks.

Deployment:  "Run this forever"
Job:         "Run this ONCE until it succeeds"
CronJob:     "Run this Job every day at 2am"

Job YAML

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  backoffLimit: 3              # retry up to 3 times
  activeDeadlineSeconds: 300   # kill after 5 minutes regardless
  ttlSecondsAfterFinished: 60  # auto-delete 60s after completion
  template:
    spec:
      restartPolicy: OnFailure  # REQUIRED — Always is invalid for Jobs
      containers:
        - name: migration
          image: mycompany/api:3.0.1
          command: ["node", "scripts/migrate.js"]

The restartPolicy: OnFailure (or Never) is mandatory — Jobs cannot use Always because they'd never complete.

Set ttlSecondsAfterFinished or finished Jobs pile up in your cluster forever.

CronJob YAML

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"        # every day at 2:00 AM
  concurrencyPolicy: Forbid    # skip if previous run still running
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 2
      ttlSecondsAfterFinished: 300
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: reporter
              image: mycompany/reporter:1.0
              command: ["python", "generate_report.py"]

Cron Schedule Quick Reference

0 2 * * *       Every day at 2:00 AM
*/5 * * * *     Every 5 minutes
0 9 * * 1       Every Monday at 9:00 AM
0 0 1 * *       1st of every month at midnight
@daily          Shortcut for 0 0 * * *
@hourly         Shortcut for 0 * * * *

Trigger a CronJob Manually

kubectl create job --from=cronjob/nightly-report manual-run-01

This creates a Job immediately using the CronJob's template, without affecting scheduled runs.

🧪 Practice — Jobs & CronJobs

Lab 1: Simple one-off Job

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: hello-job
spec:
  backoffLimit: 2
  ttlSecondsAfterFinished: 60
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: hello
          image: busybox
          command: ['sh', '-c', 'echo "Job running at $(date)"; sleep 5; echo Done!']
EOF

kubectl get pods --watch    # watch pod go: Pending → Running → Completed
kubectl logs -l job-name=hello-job
kubectl get job hello-job   # COMPLETIONS should show 1/1

Lab 2: CronJob that runs every minute

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: minute-job
spec:
  schedule: "* * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 60
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: echo
              image: busybox
              command: ['sh', '-c', 'echo "Scheduled run at $(date)"']
EOF

# Wait 1-2 minutes
kubectl get jobs
kubectl get cronjob minute-job

# Trigger it manually right now
kubectl create job --from=cronjob/minute-job manual-run-01
kubectl logs -l job-name=manual-run-01

# Suspend future runs without deleting
kubectl patch cronjob minute-job -p '{"spec":{"suspend":true}}'
kubectl get cronjob minute-job    # SUSPEND should show True

Cleanup

kubectl delete cronjob minute-job

StatefulSets: Stateful Applications

A StatefulSet is like a Deployment, but for applications that need three guarantees:

Stable, predictable names — pods are always pod-0, pod-1, pod-2
Stable storage — each pod gets its own PVC that sticks with it across restarts
Ordered startup/shutdown — pods start in order (0→1→2) and stop in reverse

DEPLOYMENT:              STATEFULSET:
my-app-abc12 (random)    mysql-0  (always 0)
my-app-def34 (random)    mysql-1  (always 1)
my-app-ghi56 (random)    mysql-2  (always 2)

Pod dies → new name      Pod dies → comes back as mysql-1
Pod dies → new IP        Pod dies → same storage re-attached
Starts in any order      Starts 0 first, then 1, then 2

The Headless Service Requirement

StatefulSets require a headless Service (clusterIP: None) to provide stable DNS names per pod:

mysql-0.mysql-svc.default.svc.cluster.local → 10.244.1.5
mysql-1.mysql-svc.default.svc.cluster.local → 10.244.2.8

A normal Service load-balances across pods. A headless Service returns individual DNS records for each pod — so applications can say "connect to the primary at mysql-0.mysql-svc" and mean it.

StatefulSet YAML

# Headless Service — create this FIRST
apiVersion: v1
kind: Service
metadata:
  name: mysql-svc
spec:
  clusterIP: None    # ← this makes it headless
  selector:
    app: mysql
  ports:
    - port: 3306
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-svc    # links to headless service above
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:     # ← key difference from Deployment
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

volumeClaimTemplates creates individual PVCs per pod: data-mysql-0, data-mysql-1, data-mysql-2.

The PVC Retention Gotcha

When you delete a StatefulSet, the PVCs are not deleted. This is intentional — data is precious. But it means they stay around until you clean them up manually:

kubectl delete sts mysql
kubectl delete pvc -l app=mysql   # must do this separately

Canary Updates with partition

The partition field in rollingUpdate is powerful for staged rollouts:

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    partition: 2    # only update pods with ordinal >= 2

Set partition: 2 to update only mysql-2 first. Test it. Lower to 1. Lower to 0 to complete. This is native canary deployment for StatefulSets.

🧪 Practice — StatefulSets

Lab: Stable identity, persistent storage, and ordered startup

# Create headless service + StatefulSet
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: web-svc
spec:
  clusterIP: None
  selector:
    app: web
  ports:
    - port: 80
      name: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: web-svc
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: www
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
EOF

# Watch ordered startup — web-0 first, then web-1, then web-2
kubectl get pods --watch

# Confirm stable names
kubectl get pods -l app=web   # always web-0, web-1, web-2

# See the per-pod PVCs created
kubectl get pvc   # www-web-0, www-web-1, www-web-2

# Write data to web-0's volume
kubectl exec web-0 -- sh -c 'echo "pod-0 data" > /usr/share/nginx/html/index.html'

# Delete web-0 — watch it come back with the SAME name
kubectl delete pod web-0
kubectl get pods --watch    # web-0 comes back, not a random name

# Data persists — PVC was reattached
kubectl exec web-0 -- cat /usr/share/nginx/html/index.html   # still "pod-0 data"

# Reach a specific pod by stable DNS from another pod
kubectl run tmp --image=busybox --rm -it --restart=Never -- \
  wget -q -O- web-0.web-svc.default.svc.cluster.local

Cleanup — note PVCs must be deleted separately

kubectl delete sts web
kubectl delete svc web-svc
kubectl delete pvc -l app=web    # PVCs are NOT auto-deleted

The Master Cheat Sheet

# LIST
kubectl get pods / deploy / rs / ds / sts / jobs / cronjobs

# USEFUL FLAGS
kubectl get pods -o wide              # +IP +NODE
kubectl get pods --show-labels        # show label columns
kubectl get pods -l app=my-app        # filter by label
kubectl get pods -A                   # all namespaces

# INSPECT (always read the Events section)
kubectl describe pod/deploy/sts <n>

# LOGS
kubectl logs <pod>
kubectl logs <pod> --previous         # after a crash
kubectl logs <pod> -c <container>     # multi-container pod

# SHELL
kubectl exec -it <pod> -- /bin/bash

# DEPLOYMENTS
kubectl scale deploy <n> --replicas=5
kubectl set image deploy/<n> <container>=<image>:tag
kubectl rollout status deploy/<n>
kubectl rollout undo deploy/<n>
kubectl rollout restart deploy/<n>

# JOBS
kubectl create job test --from=cronjob/<n>   # manual trigger
kubectl patch cronjob <n> -p '{"spec":{"suspend":true}}'

# GENERATE YAML BOILERPLATE
kubectl create deployment web --image=nginx:1.25 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

API Versions Quick Reference

Workload	apiVersion	kind
Pod	`v1`	`Pod`
Namespace	`v1`	`Namespace`
ReplicaSet	`apps/v1`	`ReplicaSet`
Deployment	`apps/v1`	`Deployment`
DaemonSet	`apps/v1`	`DaemonSet`
StatefulSet	`apps/v1`	`StatefulSet`
Job	`batch/v1`	`Job`
CronJob	`batch/v1`	`CronJob`

Restart Policy Rules

Policy	Use In	Behaviour
`Always`	Deployments, DaemonSets	Restart on any exit — default
`OnFailure`	Jobs	Restart only on non-zero exit
`Never`	Jobs	Never restart — new pod on each retry

🧪 Final Practice — End-to-End Scenario

This lab ties everything together. You'll deploy a full application stack using every concept from this article.

Scenario: Deploy a web application with a background worker, a nightly cleanup job, and per-node logging.

# Step 1: Create a dedicated namespace
kubectl create namespace myapp

# Step 2: Deploy the web app (Deployment)
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: myapp
  labels:
    app: web
    tier: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
        tier: frontend
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          resources:
            requests: { memory: "64Mi", cpu: "50m" }
            limits:   { memory: "128Mi", cpu: "200m" }
          readinessProbe:
            httpGet: { path: /, port: 80 }
            initialDelaySeconds: 3
EOF

# Step 3: Deploy a background worker (Deployment, different labels)
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
  namespace: myapp
  labels:
    app: worker
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: worker
  template:
    metadata:
      labels:
        app: worker
        tier: backend
    spec:
      containers:
        - name: worker
          image: busybox
          command: ['sh', '-c', 'while true; do echo "worker processing..."; sleep 10; done']
          resources:
            requests: { memory: "32Mi", cpu: "25m" }
EOF

# Step 4: Per-node log collector (DaemonSet)
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: myapp
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      containers:
        - name: collector
          image: busybox
          command: ['sh', '-c', 'while true; do echo "collecting logs from $(NODE_NAME)"; sleep 15; done']
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests: { memory: "20Mi", cpu: "10m" }
EOF

# Step 5: Nightly cleanup Job (CronJob)
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cleanup
  namespace: myapp
spec:
  schedule: "0 3 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 120
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: cleanup
              image: busybox
              command: ['sh', '-c', 'echo "cleaning up old data..."; sleep 3; echo done']
EOF

# Inspect the whole namespace
kubectl get all -n myapp

# Check labels
kubectl get pods -n myapp --show-labels

# Filter by tier
kubectl get pods -n myapp -l tier=frontend
kubectl get pods -n myapp -l tier=backend

# Trigger cleanup job manually
kubectl create job --from=cronjob/cleanup manual-cleanup-01 -n myapp
kubectl logs -l job-name=manual-cleanup-01 -n myapp

# Simulate rolling update on web
kubectl set image deployment/web nginx=nginx:1.26 -n myapp
kubectl rollout status deployment/web -n myapp

# Scale worker up
kubectl scale deployment worker --replicas=3 -n myapp
kubectl get pods -n myapp

Cleanup

kubectl delete namespace myapp    # deletes everything inside it

Kubernetes Foundations — Architecture, Components & kubectl

Ajinkya Singh — Sun, 22 Mar 2026 19:05:21 +0000

Why Kubernetes Exists
Cluster Architecture
Control Plane
Worker Node Components
kubectl
Local Cluster Setup
Cheat Sheet

Why Kubernetes Exists

Before Kubernetes, running containers in production meant: SSH into servers, start and stop containers by hand, write fragile shell scripts for restarts, and scale by spinning up machines one by one.

Kubernetes is a container orchestration system. It solves five problems that every team hits at scale.

🔴 Problem 1 — Single Point of Failure

Without Kubernetes:
  [App] ──► [Server A] ──✕──► App is DOWN until someone manually fixes it

With Kubernetes:
             ┌──► [Node A] ──✕── dies
  [App] ─────┤──► [Node B]  ✓  still running
             └──► [Node C]  ✓  still running
                            K8s auto-reschedules pod from A → B or C

Kubernetes detects the dead node and automatically reschedules all pods to healthy ones. No pager. No SSH.

📈 Problem 2 — Manual Scaling

Without Kubernetes:
  Traffic spike → someone notices → SSH → docker run ×10 → 20 minutes wasted

With Kubernetes:
  Traffic spike → HPA detects high CPU → pods spin up automatically → seconds, no human

The Horizontal Pod Autoscaler watches metrics and adds or removes pods automatically.

🚀 Problem 3 — Deployment Downtime

Without Kubernetes:
  Deploy v2:   stop v1 ──► [gap — users see errors] ──► start v2

With Kubernetes (rolling update):
  [v1][v1][v1]
       ↓
  [v2][v1][v1]   ← v2 started and health-checked first
       ↓
  [v2][v2][v1]
       ↓
  [v2][v2][v2]   ← v1 fully gone. Zero downtime.

New pods are health-checked before old ones are removed. Users never see an error page.

🔑 Problem 4 — Config & Secrets Chaos

Without Kubernetes:
  DB_PASSWORD="hunter2"  ← hardcoded in source code
  Different .env files per environment ← drift guaranteed

With Kubernetes:
  ConfigMaps + Secrets   ← config lives outside the image
  Same binary            ← works in dev / staging / prod

Config is decoupled from the container image entirely.

🌐 Problem 5 — Service Discovery Breakage

Without Kubernetes:
  Service A ──hardcoded──► IP 10.0.0.45 (Service B)
  Service B restarts → new IP → Service A breaks

With Kubernetes:
  Service A ──DNS──► "payment-service"
  Kubernetes routes to the right pods — IPs are irrelevant

Pods are ephemeral and get new IPs on restart. Kubernetes Services provide stable DNS names that always resolve correctly.

Common Misconceptions

Misconception	Reality
Kubernetes replaces Docker	No — K8s orchestrates containers; Docker/containerd runs them
Only for large companies	Works equally well for a 2-person startup
Cloud-only	Runs on laptops, on-prem hardware, and cloud equally
Replaces CI/CD pipelines	K8s is the deployment target, not the pipeline

Cluster Architecture

A Kubernetes cluster has two types of machines:

Control Plane — the brain. Makes all scheduling and reconciliation decisions.
Worker Nodes — the muscle. Runs your actual application containers.

All communication goes through the control plane. You never interact with worker nodes directly.

┌──────────────────────────────────────────────────────────────────────┐
│                        KUBERNETES CLUSTER                            │
│                                                                      │
│   You (kubectl)                                                      │
│        │  HTTPS                                                      │
│        ▼                                                             │
│   ┌──────────────────────────────────────────────────────────────┐  │
│   │                     CONTROL PLANE                            │  │
│   │                                                              │  │
│   │   kube-apiserver  ◄──────────────►  etcd                    │  │
│   │   (front door · port 6443)          (cluster database)      │  │
│   │                                                              │  │
│   │   kube-scheduler                  controller-manager        │  │
│   │   (pod placement)                 (reconciliation loops)    │  │
│   └───────────────┬──────────────────────────┬────────────────  ┘  │
│                   │                          │                      │
│      ┌────────────▼──────┐  ┌────────────────▼──┐  ┌────────────┐  │
│      │   Worker Node 1   │  │   Worker Node 2   │  │  Worker 3  │  │
│      │  Pod  Pod         │  │   Pod             │  │  Pod  Pod  │  │
│      │  kubelet          │  │   kubelet         │  │  kubelet   │  │
│      │  kube-proxy       │  │   kube-proxy      │  │  kube-proxy│  │
│      │  containerd       │  │   containerd      │  │  containerd│  │
│      └───────────────────┘  └───────────────────┘  └────────────┘  │
└──────────────────────────────────────────────────────────────────────┘

▶ Full flow: kubectl create deployment → running container

1.   kubectl          reads ~/.kube/config → finds API server URL
2.   kube-apiserver   authenticates, RBAC check, validates request
3.   kube-apiserver   writes Deployment object to etcd
4.   controller-mgr   detects new Deployment → creates ReplicaSet → unscheduled Pod
5.   kube-scheduler   filters + scores nodes → binds pod to best node
6.   kubelet          sees assignment → calls containerd via CRI
7.   containerd       pulls image → calls runc → starts container
8.   kubelet          reports "Running" to API server
9.   etcd             pod status updated
10.  kubectl get pods → my-app  1/1  Running ✓

Control Plane

kube-apiserver

The single entry point for everything — kubectl, CI/CD pipelines, dashboards, and internal components. Validates requests, enforces RBAC, persists state to etcd. Port 6443 over HTTPS.

▶ The 4-gate request pipeline

Incoming request
       │
       ▼
┌─ GATE 1: AUTHENTICATION ─────────────────────────────────────────┐
│  "Who are you?"                                                  │
│  TLS client certificates · Bearer tokens · OIDC                 │
│  Failure → 401 Unauthorized                                      │
└──────────────────────────────────────┬───────────────────────────┘
                                       │
                                       ▼
┌─ GATE 2: AUTHORIZATION (RBAC) ───────────────────────────────────┐
│  "Are you allowed to do this?"                                   │
│  Checks Roles and ClusterRoles bound to the user                 │
│  Failure → 403 Forbidden                                         │
└──────────────────────────────────────┬───────────────────────────┘
                                       │
                                       ▼
┌─ GATE 3: ADMISSION CONTROL ──────────────────────────────────────┐
│  Mutating webhooks   → can modify the request                    │
│  Validating webhooks → can reject the request                    │
│  Failure → 400 / 422                                             │
└──────────────────────────────────────┬───────────────────────────┘
                                       │
                                       ▼
┌─ GATE 4: PERSIST TO etcd ────────────────────────────────────────┐
│  Object written → 201 Created returned ✓                         │
└──────────────────────────────────────────────────────────────────┘

Controllers and kubectl --watch use long-lived HTTP watch streams, not polling:

# GET /api/v1/namespaces/default/pods?watch=true
# Server streams:
# ADDED    my-pod   Pending
# MODIFIED my-pod   Running
# DELETED  my-pod

etcd

Distributed key-value store. The single source of truth for all cluster state. Losing etcd without a backup = losing all cluster configuration.

/registry/pods/default/my-pod         →  { pod spec }
/registry/deployments/default/my-app  →  { deployment spec }
/registry/services/default/my-svc     →  { service config }

etcd uses Raft consensus — a write commits only when a majority quorum agrees.

▶ Raft quorum explained

3-node etcd cluster:

  Leader ──heartbeat──► Follower 1
         ──heartbeat──► Follower 2

Write committed when majority (2 of 3) agree ✓
Leader + one Follower offline → 1 of 3 → no majority → writes halt ✗

Cluster Size	Quorum	Failures Tolerated
1	1	0
3	2	1 ← production minimum
5	3	2 ← recommended
7	4	3

Always use an odd number. Even numbers risk a 50/50 split → no quorum → all writes halt.

kube-scheduler

Decides which node a new pod runs on. Does not start the pod — only decides placement.

▶ How the scheduler picks a node

New pod (unscheduled)
       │
       ▼
PHASE 1 — FILTER: "Which nodes can run this pod?"
  Node 1: 90% CPU → skip
  Node 2: 20% CPU → feasible ✓
  Node 3: 55% CPU → feasible ✓

PHASE 2 — SCORE: "Which is the best node?"
  Node 2: score 85  (less loaded, image already cached)
  Node 3: score 62

PHASE 3 — BIND:
  Pod assigned to Node 2 → kubelet starts the container

Diagnosing a pod stuck in Pending:

kubectl describe pod my-pod
# Events:
# Warning  FailedScheduling  0/3 nodes are available:
#   3 Insufficient memory.

controller-manager

Runs all built-in controllers in a single binary. Each one watches for drift between desired and actual state and corrects it — forever.

while True:
    desired = api_server.get_desired_state()
    actual  = api_server.get_actual_state()
    if desired != actual:
        take_corrective_action()   # create / delete / update
    sleep(brief_interval)

▶ Built-in controllers

Controller	Responsibility
Deployment Controller	Manages ReplicaSets to maintain desired replicas
ReplicaSet Controller	Ensures the correct number of Pods exist
Node Controller	Detects node failures, evicts pods from dead nodes
Job Controller	Runs pods to completion for batch workloads
CronJob Controller	Creates Jobs on a cron schedule
Endpoint Controller	Updates Service endpoints when backing pods change
Namespace Controller	Cleans up resources when a namespace is deleted

Production note: Run 3 or 5 control plane nodes for HA. Back up etcd on a schedule. Never run user workloads on control plane nodes.

Worker Node Components

kubelet

Runs on every worker node as a systemd service — not a Kubernetes pod. If it crashes, systemd restarts it.

▶ What kubelet manages

Responsibility	Detail
PodSpecs	Downloads pod definitions from the API server
Container lifecycle	Start, stop, restart via the container runtime
Health checks	Runs liveness, readiness, and startup probes
Resource reporting	Reports CPU/memory usage to the API server
Volume mounting	Mounts ConfigMaps, Secrets, and PVCs into containers
Static pods	Reads YAML from `/etc/kubernetes/manifests/` directly

▶ Static pods — how the control plane bootstraps itself

Static pods allow kubelet to start containers without a running API server, by reading YAML files from disk. This is how K8s starts itself from scratch:

/etc/kubernetes/manifests/
  etcd.yaml
  kube-apiserver.yaml
  kube-controller-manager.yaml
  kube-scheduler.yaml

Bootstrap order:
  systemd starts kubelet
    → kubelet reads manifests → starts etcd
    → starts kube-apiserver
    → everything else connects normally

# Inspect in a kind cluster:
docker exec kind-control-plane ls /etc/kubernetes/manifests/

kube-proxy

Programs iptables or IPVS rules into the Linux kernel for Service routing. Not in the packet data path at runtime — it writes the rules and steps aside.

Client → GET http://my-service:80
       ↓
Linux kernel intercepts (rules written by kube-proxy)
       ├── 33% ──► Pod 10.244.1.5 (Node 1)
       ├── 33% ──► Pod 10.244.2.8 (Node 2)
       └── 34% ──► Pod 10.244.3.2 (Node 3)

▶ Proxy modes comparison

Mode	Mechanism	Lookup	Use Case
`iptables` (default)	Linux iptables chains	O(n)	< ~1,000 Services
`ipvs`	Kernel hash table	O(1)	Large clusters
eBPF (Cilium)	Bypasses iptables entirely	Best-in-class	Performance-critical

containerd

The default container runtime. kubelet calls it via the CRI (Container Runtime Interface).

kubelet → (gRPC / CRI) → containerd → (OCI) → runc → container

▶ The "Docker was removed" myth — clarified

What was removed in K8s 1.24: dockershim (an adapter between kubelet and Docker's API).

What was NOT removed:
  ✓  Docker images and Dockerfiles → still work perfectly
  ✓  docker build                  → still valid locally

Before K8s 1.24:  kubelet → dockershim → Docker → containerd → runc
After  K8s 1.24:  kubelet → containerd → runc

Docker was a middleman. containerd — which Docker itself uses internally — always did the actual work.

Runtime	Used By	Notes
containerd	GKE, EKS, AKS, kind	Default — lightweight, fast
CRI-O	OpenShift, bare-metal	Built specifically for Kubernetes

kubectl

kubectl reads ~/.kube/config to find your cluster's API server URL and credentials, then talks to kube-apiserver over HTTPS.

Managing Multiple Clusters

kubectl config view
kubectl config get-contexts
kubectl config use-context kind-k8s-lab
kubectl config current-context
kubectl config set-context --current --namespace=my-team
kubectl get pods --context=production-cluster

Command Structure

kubectl  [verb]    [resource]   [name]      [flags]
         get       pods         my-pod      -n production
         describe  deployment   my-dep
         delete    service      my-svc
         apply                  -f app.yaml
         exec      -it my-pod   --          /bin/bash
         logs      my-pod                   -f --previous

▶ Getting information

kubectl get pods
kubectl get pods -n kube-system        # specific namespace
kubectl get pods -A                    # all namespaces
kubectl get pods -o wide               # with IP and NODE columns
kubectl get pods -o yaml               # full YAML spec
kubectl get pods --watch               # live stream
kubectl get pods -l app=my-app         # filter by label
kubectl get all -n my-namespace

kubectl describe pod my-pod            # full details + Events ← debug here
kubectl describe node my-node
kubectl describe deployment my-dep

> Always read the Events section in kubectl describe — it is the primary debugging signal in Kubernetes.

▶ Creating and updating resources

kubectl apply -f app.yaml              # create or update (idempotent) ← always prefer this
kubectl apply -f ./manifests/
kubectl create deployment my-app --image=nginx
kubectl create namespace my-team

kubectl apply is idempotent — creates if absent, updates if present. kubectl create fails if the resource already exists. Use apply in all scripts and pipelines.

▶ Deleting resources

kubectl delete pod my-pod
kubectl delete -f app.yaml
kubectl delete deployment my-app                        # removes ReplicaSet and pods too
kubectl delete pod my-pod --force --grace-period=0      # immediate (stuck pods)

▶ Logs

kubectl logs my-pod
kubectl logs my-pod -f                      # stream live
kubectl logs my-pod --previous              # logs before last restart ← use after crashes
kubectl logs my-pod -c my-container         # specific container
kubectl logs my-pod --tail=100
kubectl logs my-pod --since=1h
kubectl logs -l app=my-app                  # all matching pods

▶ Exec into a pod

kubectl exec -it my-pod -- /bin/bash
kubectl exec -it my-pod -- /bin/sh          # if bash is unavailable
kubectl exec my-pod -- ls /app
kubectl exec -it my-pod -c sidecar -- /bin/sh

# Ephemeral debug pod — auto-deleted on exit
kubectl run debug --image=busybox --rm -it --restart=Never -- /bin/sh

▶ Port forwarding

kubectl port-forward pod/my-pod 8080:80
kubectl port-forward service/my-svc 8080:80
kubectl port-forward deployment/my-app 8080:3000

▶ Scaling and rollouts

kubectl scale deployment my-app --replicas=5
kubectl set image deployment/my-app nginx=nginx:1.25
kubectl rollout status deployment/my-app
kubectl rollout history deployment/my-app
kubectl rollout undo deployment/my-app
kubectl rollout undo deployment/my-app --to-revision=3

▶ JSONPath — extract specific fields

kubectl get pod my-pod -o jsonpath='{.status.podIP}'
kubectl get pods -o jsonpath='{.items[*].metadata.name}'

# All pods: namespace, name, IP
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'

Common Mistakes

Mistake	Correct Approach
Using `kubectl create` in scripts	Use `kubectl apply` — idempotent
Forgetting `-n <namespace>`	Use `-A` or set a default namespace in the context
Deleting a pod and expecting it to stay gone	Delete the Deployment — the ReplicaSet recreates pods
Skipping the Events section in `describe`	Events are the primary debugging signal
Not using `--previous` after a crash	`kubectl logs my-pod --previous` shows pre-crash logs

Local Cluster Setup

▶ Tool comparison

Tool	Nodes	Use Case
kind (recommended)	Multi-node	CI/CD, multi-node testing, closest to production
minikube	Single-node	Quick experiments, rich addons

Install

# macOS
brew install kubectl kind

# Linux — kubectl
curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# Linux — kind
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
chmod +x kind && sudo mv kind /usr/local/bin/

Create a Multi-Node Cluster

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: k8s-lab
nodes:
  - role: control-plane
  - role: worker
  - role: worker

kind create cluster --config kind-config.yaml

kubectl get nodes
# k8s-lab-control-plane  Ready  control-plane
# k8s-lab-worker         Ready  <none>
# k8s-lab-worker2        Ready  <none>

Verify Health

kubectl get nodes                   # all nodes → Ready
kubectl get pods -n kube-system     # all system pods → Running
kubectl cluster-info

▶ First application lab

# Deploy
kubectl create deployment my-nginx --image=nginx:1.25 --replicas=2
kubectl get pods -o wide

# Expose
kubectl expose deployment my-nginx --port=80 --name=my-nginx-service
kubectl describe service my-nginx-service

# Access
kubectl port-forward service/my-nginx-service 8080:80 &amp;
curl http://localhost:8080

# Scale
kubectl scale deployment my-nginx --replicas=4

# Rolling update
kubectl set image deployment/my-nginx nginx=nginx:1.26
kubectl rollout status deployment/my-nginx

# Rollback
kubectl rollout undo deployment/my-nginx

# Clean up
kill %1
kubectl delete deployment my-nginx
kubectl delete service my-nginx-service

kind Management

kind get clusters
kind create cluster --config kind-config.yaml
kind delete cluster --name k8s-lab
kind load docker-image my-app:latest --name k8s-lab
kubectl config use-context kind-k8s-lab

Cheat Sheet

▶ Architecture & key concepts

Control Plane:   kube-apiserver · etcd · kube-scheduler · controller-manager
Worker Nodes:    kubelet · kube-proxy · containerd
Full flow:       kubectl → apiserver → etcd/scheduler → kubelet → container

Reconciliation:  desired ≠ actual → controller fixes it → repeat forever
Static pods:     /etc/kubernetes/manifests/ → kubelet reads directly
kubeconfig:      ~/.kube/config → cluster URLs + credentials
etcd quorum:     always odd — 3 or 5 in production
Scheduler:       filter → score → bind to winner
kubelet:         systemd service (not a pod); bootstraps the control plane

▶ kubectl quick reference

# Get
kubectl get pods / svc / deploy / nodes
kubectl get pods -A                      # all namespaces
kubectl get pods -o wide                 # +IP +NODE
kubectl get pod  -o yaml             # full spec
kubectl get pods --watch                 # live stream
kubectl describe pod                # details + Events ← debug here
kubectl explain pod.spec.containers      # built-in docs

# Apply / Delete
kubectl apply -f file.yaml
kubectl delete -f file.yaml
kubectl delete pod  --force

# Logs
kubectl logs  -f
kubectl logs  --previous            # after crash ← important
kubectl logs  -c 
kubectl logs -l app=my-app

# Exec / Port-forward
kubectl exec -it  -- /bin/bash
kubectl port-forward svc/ 8080:80

# Scale / Rollout
kubectl scale deployment  --replicas=5
kubectl set image deployment/ =<img>:tag
kubectl rollout status deployment/
kubectl rollout history deployment/
kubectl rollout undo deployment/

# Context
kubectl config get-contexts
kubectl config use-context 
kubectl config set-context --current --namespace=

▶ kind quick reference

kind create cluster --config kind-config.yaml
kind get clusters
kind load docker-image  --name 
kind delete cluster --name

🤖 AI-generated · Personal reference only · Not original content"

📚 Complete Java Microservices Guide: Dropwizard & Ecosystem

Ajinkya Singh — Thu, 26 Feb 2026 17:39:46 +0000

A Step-by-Step Beginner-to-Intermediate Learning Path

How to use this guide: Read chapter by chapter. Every concept builds on the previous one. Code examples are practical and runnable. Chapters marked 🔲 are TODO — topics to study next.

📋 Table of Contents

#	Chapter	Status
1	What is Dropwizard & Why Use It?	✅
2	Project Setup & Fat JAR	✅
3	Configuration with YAML	✅
4	Building REST Resources (JAX-RS)	✅
5	Bean Validation (Hibernate Validator)	✅
6	Jackson – JSON Serialization	✅
7	Database Access with Hibernate/JDBI	✅
8	Liquibase – Database Migrations	✅
9	Authentication (Basic Auth & OAuth2)	✅
10	Health Checks & Metrics	✅
11	Testing in Dropwizard	✅
12	Lombok – Boilerplate Killer	✅
13	MapStruct – Object Mapping	✅
14	Google Guice – Dependency Injection	✅
15	OkHttp – HTTP Client	✅
16	Hystrix – Circuit Breaker	✅
17	TODO: Advanced Topics	🔲

Chapter 1: What is Dropwizard & Why Use It?

🤔 The Problem It Solves

Imagine you want to build a Java REST API. You'd need to:

Pick a web server (Jetty, Tomcat?)
Pick a JSON library (Jackson, Gson?)
Pick a validation library
Pick a database library
Pick a metrics/monitoring library
Make sure they all work together (version conflicts are a nightmare!)

This research alone can take weeks and paralyze a developer.

✅ Dropwizard's Solution

Dropwizard is a curated collection of best-of-breed libraries glued together so they work out of the box.

Component	Library Used
Web Server	Jetty (embedded)
REST Framework	Jersey (JAX-RS)
JSON	Jackson
Validation	Hibernate Validator
Database Migrations	Liquibase
Metrics & Health	Metrics library
Logging	Logback + SLF4J
Config	YAML + Jackson

🫙 The Fat JAR Concept

Traditional Java apps rely on an app server (Tomcat, GlassFish) being installed on the machine. The problem:

"It works on my computer" 😤

Dropwizard packages everything — your code + all dependencies + the web server — into a single JAR file.

myapp.jar  ← Contains Jetty, Jersey, Jackson, your code... everything!

To run it:

java -jar myapp.jar server config.yml

That's it. No app server needed. No "works on my machine" problem.

🏗️ Microservices Context

Microservices = breaking a big monolithic app into small, independent services. Each service:

Does one thing well
Has its own database
Communicates via REST/HTTP
Can be deployed independently

Dropwizard is perfect for microservices because each service is a single JAR that starts up fast.

🆚 Dropwizard vs Spring Boot

Feature	Dropwizard	Spring Boot
Philosophy	Opinionated, curated	Flexible, extensive
Learning curve	Easier if you know JAX-RS	Easier if you know Spring
Startup speed	Fast	Moderate
Ecosystem size	Smaller	Huge
Best for	REST APIs, microservices	Anything Java

Which to choose? If you're building a focused REST microservice, Dropwizard is excellent. For complex enterprise apps, Spring Boot has more features.

Chapter 2: Project Setup & Fat JAR

🛠️ Prerequisites

Make sure you have installed:

Java 8+ (Dropwizard 1.x requires Java 8)
Maven 3.x
An IDE (IntelliJ IDEA recommended)

📁 Creating a Project with Maven Archetype

The fastest way to start:

mvn archetype:generate \
  -DarchetypeGroupId=io.dropwizard.archetypes \
  -DarchetypeArtifactId=java-simple \
  -DarchetypeVersion=1.3.29 \
  -DgroupId=com.example \
  -DartifactId=my-api \
  -Dversion=1.0-SNAPSHOT \
  -Dname=MyApi

⚠️ Important: The -Dname=MyApi parameter is used in generated file names. Always provide it!

📂 Generated Project Structure

my-api/
├── pom.xml
└── src/
    └── main/
        ├── java/
        │   └── com/example/
        │       ├── MyApiApplication.java   ← Main class (entry point)
        │       └── MyApiConfiguration.java ← Config class
        └── resources/
            └── config.yml                  ← Configuration file

📄 The `pom.xml` — Your Project's Heart

<project>
    <groupId>com.example</groupId>
    <artifactId>my-api</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <dropwizard.version>1.3.29</dropwizard.version>
    </properties>

    <dependencies>
        <!-- Core Dropwizard -->
        <dependency>
            <groupId>io.dropwizard</groupId>
            <artifactId>dropwizard-core</artifactId>
            <version>${dropwizard.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!-- This plugin creates the Fat JAR -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <configuration>
                    <createDependencyReducedPom>true</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals><goal>shade</goal></goals>
                        <configuration>
                            <transformers>
                                <transformer implementation=
                                  "org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                                <transformer implementation=
                                  "org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <!-- This tells Java which class has main() -->
                                    <mainClass>com.example.MyApiApplication</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

🚀 The Application Class

package com.example;

import io.dropwizard.Application;
import io.dropwizard.setup.Bootstrap;
import io.dropwizard.setup.Environment;

public class MyApiApplication extends Application<MyApiConfiguration> {

    // Java's main method — this starts everything
    public static void main(String[] args) throws Exception {
        new MyApiApplication().run(args);
    }

    @Override
    public String getName() {
        return "my-api";  // Name of your app
    }

    @Override
    public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
        // Register bundles (plugins) here
    }

    @Override
    public void run(MyApiConfiguration configuration, Environment environment) {
        // Register your resources (REST endpoints) here
        // This is where the app "comes alive"
    }
}

🏗️ Building & Running

# Build the Fat JAR
mvn clean package

# Run with server command
java -jar target/my-api-1.0-SNAPSHOT.jar server config.yml

# Check if it's alive (admin port 8081)
curl http://localhost:8081/ping

# Your API is on port 8080 by default
curl http://localhost:8080/hello

📋 Built-in Commands

# Validate your config file (catches errors before running!)
java -jar my-api.jar check config.yml

# Run the server
java -jar my-api.jar server config.yml

# Database commands (when Liquibase is added)
java -jar my-api.jar db migrate config.yml
java -jar my-api.jar db status config.yml
java -jar my-api.jar db drop-all --confirm-delete-everything config.yml

💡 Pro tip: Run check before server in deployment scripts to catch config errors early!

Chapter 3: Configuration with YAML

🤷 Why Configuration Matters

Hard-coding things like database URLs, passwords, and ports in code is bad practice. Configuration files let you:

Have different settings for dev/test/production
Change settings without recompiling
Keep secrets out of code

📄 The `config.yml` File

# Server ports
server:
  applicationConnectors:
    - type: http
      port: 8080
  adminConnectors:
    - type: http
      port: 8081

# Logging
logging:
  level: INFO
  loggers:
    com.example: DEBUG  # More verbose logging for your package

# Custom config (your own properties)
login: admin
password: secret123

# Database (added later with Hibernate bundle)
database:
  driverClass: com.mysql.jdbc.Driver
  url: jdbc:mysql://localhost:3306/mydb
  user: dbuser
  password: dbpassword

☕ The Configuration Class

Every key in config.yml maps to a field in your Configuration class:

package com.example;

import io.dropwizard.Configuration;
import com.fasterxml.jackson.annotation.JsonProperty;
import org.hibernate.validator.constraints.NotEmpty;

public class MyApiConfiguration extends Configuration {

    // @NotEmpty = Hibernate Validator annotation
    // If this field is empty/missing, app won't start!
    @NotEmpty
    private String login;

    @NotEmpty
    private String password;

    // @JsonProperty tells Jackson the YAML key name
    @JsonProperty
    public String getLogin() {
        return login;
    }

    @JsonProperty
    public void setLogin(String login) {
        this.login = login;
    }

    @JsonProperty
    public String getPassword() {
        return password;
    }

    @JsonProperty
    public void setPassword(String password) {
        this.password = password;
    }
}

🌍 Multiple Config Files (Best Practice)

# Development
java -jar my-api.jar server config-dev.yml

# Production  
java -jar my-api.jar server config-prod.yml

# Testing (uses H2 in-memory database)
java -jar my-api.jar server config-test.yml

Each config file can point to different databases, use different ports, etc.

🔐 Accessing Config in Your App

@Override
public void run(MyApiConfiguration config, Environment environment) {
    // Access config values
    String login = config.getLogin();
    String password = config.getPassword();

    // Pass them to classes that need them
    environment.jersey().register(new MyResource(login, password));
}

Chapter 4: Building REST Resources (JAX-RS)

🌐 What is JAX-RS?

JAX-RS is a Java standard (specification) for building REST APIs using annotations. Dropwizard uses Jersey as the JAX-RS implementation.

The key idea: annotate a regular Java class with special annotations, and Jersey turns it into a REST endpoint.

📌 Core Annotations

Annotation	Meaning
`@Path("/users")`	URL path this class/method handles
`@GET`	Responds to HTTP GET requests
`@POST`	Responds to HTTP POST requests
`@PUT`	Responds to HTTP PUT requests
`@DELETE`	Responds to HTTP DELETE requests
`@Produces(MediaType.APPLICATION_JSON)`	Returns JSON
`@Consumes(MediaType.APPLICATION_JSON)`	Accepts JSON input
`@PathParam("id")`	Gets value from URL: `/users/42` → id=42
`@QueryParam("name")`	Gets value from query string: `?name=John`

☕ Your First Resource Class

package com.example.resources;

import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.List;
import java.util.ArrayList;

@Path("/bookmarks")         // All methods in this class handle /bookmarks
@Produces(MediaType.APPLICATION_JSON)  // All responses are JSON
@Consumes(MediaType.APPLICATION_JSON)  // All requests accept JSON
public class BookmarkResource {

    private List<Bookmark> bookmarks = new ArrayList<>();

    // GET /bookmarks → returns all bookmarks
    @GET
    public List<Bookmark> getAllBookmarks() {
        return bookmarks;
    }

    // GET /bookmarks/42 → returns one bookmark by ID
    @GET
    @Path("/{id}")
    public Bookmark getBookmark(@PathParam("id") long id) {
        return bookmarks.stream()
            .filter(b -> b.getId() == id)
            .findFirst()
            .orElseThrow(() -> new NotFoundException("Bookmark not found: " + id));
    }

    // POST /bookmarks → create a new bookmark
    @POST
    public Response createBookmark(Bookmark bookmark) {
        bookmarks.add(bookmark);
        return Response.status(Response.Status.CREATED)
                       .entity(bookmark)
                       .build();
    }

    // DELETE /bookmarks/42 → delete a bookmark
    @DELETE
    @Path("/{id}")
    public Response deleteBookmark(@PathParam("id") long id) {
        bookmarks.removeIf(b -> b.getId() == id);
        return Response.noContent().build();  // 204 No Content
    }

    // GET /bookmarks/search?tag=java → filter by query param
    @GET
    @Path("/search")
    public List<Bookmark> searchByTag(@QueryParam("tag") String tag) {
        // filter logic here
        return bookmarks;
    }
}

📦 The Domain Model (Bookmark.java)

package com.example.core;

public class Bookmark {
    private long id;
    private String url;
    private String title;
    private String tag;

    // Default constructor (required for JSON deserialization)
    public Bookmark() {}

    public Bookmark(long id, String url, String title) {
        this.id = id;
        this.url = url;
        this.title = title;
    }

    // Getters and setters
    public long getId() { return id; }
    public void setId(long id) { this.id = id; }
    public String getUrl() { return url; }
    public void setUrl(String url) { this.url = url; }
    public String getTitle() { return title; }
    public void setTitle(String title) { this.title = title; }
    public String getTag() { return tag; }
    public void setTag(String tag) { this.tag = tag; }
}

📝 Register the Resource in Application

@Override
public void run(MyApiConfiguration config, Environment environment) {
    // Without this line, Dropwizard doesn't know about your resource!
    environment.jersey().register(new BookmarkResource());
}

🔧 HTTP Status Codes Quick Reference

Code	Meaning	When to use
200	OK	Successful GET, PUT
201	Created	Successful POST (new resource created)
204	No Content	Successful DELETE
400	Bad Request	Invalid input
401	Unauthorized	Not authenticated
403	Forbidden	Authenticated but no permission
404	Not Found	Resource doesn't exist
500	Server Error	Something blew up

🧑‍💻 Recommended Folder Structure

src/main/java/com/example/
├── MyApiApplication.java       ← Entry point
├── MyApiConfiguration.java     ← Config class
├── core/                       ← Domain models / entities
│   └── Bookmark.java
├── db/                         ← Database access (DAOs)
│   └── BookmarkDAO.java
├── resources/                  ← REST endpoints
│   └── BookmarkResource.java
└── auth/                       ← Authentication
    └── MyAuthenticator.java

Chapter 5: Bean Validation (Hibernate Validator)

❓ Why Validate?

Users send bad data. Always. Never trust input. Validation catches bad data before it hits your database.

Without validation:

{ "url": "", "title": null }

This would save garbage data to your database!

📦 Dependency (already included in dropwizard-core)

Dropwizard includes Hibernate Validator automatically. No extra dependency needed!

🏷️ Common Validation Annotations

import javax.validation.constraints.*;
import org.hibernate.validator.constraints.*;

public class Bookmark {

    private long id;

    @NotEmpty(message = "URL cannot be empty")
    @URL(message = "Must be a valid URL")
    private String url;

    @NotEmpty
    @Size(min = 1, max = 200, message = "Title must be 1-200 characters")
    private String title;

    @NotNull
    private String tag;

    @Min(value = 0, message = "Views cannot be negative")
    private int views;

    @Email(message = "Must be a valid email")
    private String authorEmail;

    @Pattern(regexp = "^[A-Z].*", message = "Must start with uppercase")
    private String category;
}

✅ Using `@Valid` in Resources

Add @Valid before the parameter — Dropwizard validates automatically!

@POST
public Response createBookmark(@Valid Bookmark bookmark) {
    // If bookmark fails validation, Dropwizard returns 422 Unprocessable Entity
    // with details about what's wrong — automatically!
    bookmarks.add(bookmark);
    return Response.status(201).entity(bookmark).build();
}

If validation fails, the client gets:

{
  "errors": [
    "bookmark.url may not be empty",
    "bookmark.title size must be between 1 and 200"
  ]
}

🛡️ Validating Config Too!

public class MyApiConfiguration extends Configuration {

    @NotEmpty  // App won't start if login is missing in config.yml!
    @JsonProperty
    private String login;

    @NotEmpty
    @JsonProperty
    private String password;
}

✍️ Custom Validator

// 1. Create the annotation
@Documented
@Constraint(validatedBy = NoSpacesValidator.class)
@Target({ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
public @interface NoSpaces {
    String message() default "Must not contain spaces";
    Class<?>[] groups() default {};
    Class<? extends Payload>[] payload() default {};
}

// 2. Create the validator logic
public class NoSpacesValidator implements ConstraintValidator<NoSpaces, String> {
    @Override
    public boolean isValid(String value, ConstraintValidatorContext ctx) {
        return value == null || !value.contains(" ");
    }
}

// 3. Use it
public class Bookmark {
    @NoSpaces
    private String slug;
}

Chapter 6: Jackson – JSON Serialization

🔄 What is Jackson?

Jackson converts between Java objects ↔ JSON. This is called serialization (Java → JSON) and deserialization (JSON → Java).

Dropwizard uses Jackson automatically for all REST responses. When your method returns a Bookmark object, Jackson converts it to JSON for you.

📦 Dependency

Already included in dropwizard-core!

🏷️ Key Jackson Annotations

import com.fasterxml.jackson.annotation.*;

public class Bookmark {

    // Changes the JSON key name
    @JsonProperty("bookmark_id")
    private long id;  // Java field "id" → JSON key "bookmark_id"

    // Include in JSON even if null
    @JsonInclude(JsonInclude.Include.ALWAYS)
    private String title;

    // Exclude from JSON output
    @JsonIgnore
    private String internalNotes;

    // Exclude null values from JSON
    @JsonInclude(JsonInclude.Include.NON_NULL)
    private String optionalField;

    // Custom date format
    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss")
    private Date createdAt;

    // Map JSON → object (deserialization constructor)
    @JsonCreator
    public Bookmark(@JsonProperty("bookmark_id") long id,
                    @JsonProperty("url") String url) {
        this.id = id;
        this.url = url;
    }
}

🗺️ Working with Unknown Fields

// Ignore any extra JSON fields you don't have a field for
@JsonIgnoreProperties(ignoreUnknown = true)
public class Bookmark {
    private String url;
    // JSON might have "extra_field" — this annotation ignores it instead of throwing an error
}

⚙️ Customizing Jackson in Dropwizard

@Override
public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
    // Customize the ObjectMapper
    bootstrap.getObjectMapper()
        .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
        .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
        .setSerializationInclusion(JsonInclude.Include.NON_NULL);
}

🌐 Polymorphic Types (Advanced)

// Base class
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes({
    @JsonSubTypes.Type(value = WebBookmark.class, name = "web"),
    @JsonSubTypes.Type(value = BookBookmark.class, name = "book")
})
public abstract class Bookmark { }

public class WebBookmark extends Bookmark { private String url; }
public class BookBookmark extends Bookmark { private String isbn; }

// JSON will look like:
// { "type": "web", "url": "https://..." }
// { "type": "book", "isbn": "978-..." }

Chapter 7: Database Access with Hibernate/JDBI

🗃️ Your Two Options in Dropwizard

Option	Description	Best For
Hibernate (JPA)	ORM — maps Java objects to DB tables	Complex domain models
JDBI	Thin wrapper over JDBC — write SQL directly	Simple, fast queries

📦 Dependencies

<!-- For Hibernate -->
<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-hibernate</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

<!-- MySQL Driver (use 5.x to avoid Liquibase conflicts!) -->
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>5.1.49</version>
</dependency>

🗄️ Part A: Hibernate

Step 1 — The Entity (maps to a DB table)

package com.example.core;

import javax.persistence.*;

@Entity                         // This Java class = a database table
@Table(name = "bookmarks")      // Table name in the database
@NamedQueries({
    // Pre-defined queries attached to this class
    @NamedQuery(
        name = "com.example.core.Bookmark.findAll",
        query = "SELECT b FROM Bookmark b"     // JPQL (not SQL!)
    ),
    @NamedQuery(
        name = "com.example.core.Bookmark.findByTag",
        query = "SELECT b FROM Bookmark b WHERE b.tag = :tag"
    )
})
public class Bookmark {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)  // Auto-increment
    private long id;

    @Column(name = "url", nullable = false)
    private String url;

    @Column(name = "title")
    private String title;

    @Column(name = "tag")
    private String tag;

    // Getters, setters, default constructor...
}

Step 2 — The DAO (Data Access Object)

package com.example.db;

import io.dropwizard.hibernate.AbstractDAO;
import com.example.core.Bookmark;
import org.hibernate.SessionFactory;
import org.hibernate.query.Query;
import java.util.List;
import java.util.Optional;

public class BookmarkDAO extends AbstractDAO<Bookmark> {

    public BookmarkDAO(SessionFactory factory) {
        super(factory);  // AbstractDAO handles Hibernate session management
    }

    // Get all bookmarks
    public List<Bookmark> findAll() {
        return list(namedQuery("com.example.core.Bookmark.findAll"));
    }

    // Get by ID — returns Optional (empty if not found)
    public Optional<Bookmark> findById(long id) {
        return Optional.ofNullable(get(id));
    }

    // Get by tag — using named query parameter
    public List<Bookmark> findByTag(String tag) {
        Query<Bookmark> query = namedQuery("com.example.core.Bookmark.findByTag");
        query.setParameter("tag", tag);
        return list(query);
    }

    // Create or update
    public Bookmark save(Bookmark bookmark) {
        return persist(bookmark);  // AbstractDAO's persist = INSERT or UPDATE
    }

    // Delete
    public void delete(Bookmark bookmark) {
        currentSession().delete(bookmark);
    }
}

Step 3 — Register Hibernate Bundle in Application

public class MyApiApplication extends Application<MyApiConfiguration> {

    // Create the Hibernate bundle — tell it about all your entities!
    private final HibernateBundle<MyApiConfiguration> hibernate =
        new HibernateBundle<MyApiConfiguration>(Bookmark.class) {  // list ALL entities here
            @Override
            public DataSourceFactory getDataSourceFactory(MyApiConfiguration config) {
                return config.getDataSourceFactory();
            }
        };

    @Override
    public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
        bootstrap.addBundle(hibernate);  // Register the bundle!
    }

    @Override
    public void run(MyApiConfiguration config, Environment environment) {
        // Create DAO with Hibernate session factory
        final BookmarkDAO dao = new BookmarkDAO(hibernate.getSessionFactory());

        // Pass DAO to resource
        environment.jersey().register(new BookmarkResource(dao));
    }
}

Step 4 — Config file addition

# config.yml
database:
  driverClass: com.mysql.jdbc.Driver
  url: jdbc:mysql://localhost:3306/bookmarks_db
  user: root
  password: secret
  properties:
    charSet: UTF-8
    hibernate.dialect: org.hibernate.dialect.MySQL5InnoDBDialect
  maxWaitForConnection: 1s
  validationQuery: "/* MyService Health Check */ SELECT 1"
  minSize: 2
  maxSize: 8
  checkConnectionWhileIdle: false

Step 5 — Add DataSourceFactory to Config class

public class MyApiConfiguration extends Configuration {

    @Valid
    @NotNull
    private DataSourceFactory database = new DataSourceFactory();

    @JsonProperty("database")
    public DataSourceFactory getDataSourceFactory() { return database; }

    @JsonProperty("database")
    public void setDataSourceFactory(DataSourceFactory factory) {
        this.database = factory;
    }
}

Step 6 — Use `@UnitOfWork` in Resource

public class BookmarkResource {
    private final BookmarkDAO dao;

    public BookmarkResource(BookmarkDAO dao) {
        this.dao = dao;
    }

    @GET
    // @UnitOfWork manages the Hibernate session + transaction for you!
    // Without it, you'd have to open/close sessions manually.
    @UnitOfWork
    public List<Bookmark> getAllBookmarks() {
        return dao.findAll();
        // ⚠️ WARNING: Lazy-loaded fields must be accessed INSIDE this method!
    }

    @POST
    @UnitOfWork
    public Bookmark createBookmark(@Valid Bookmark bookmark) {
        return dao.save(bookmark);
    }
}

Chapter 8: Liquibase – Database Migrations

🤔 The Problem Without Migrations

Imagine two developers both changing the database schema. Developer A adds a tag column, Developer B renames url to link. Who goes first? How do you track DB changes? How do you roll back?

Liquibase is like Git, but for your database schema.

📦 Dependency

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-migrations</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

⚙️ Register the Bundle

@Override
public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
    bootstrap.addBundle(new MigrationsBundle<MyApiConfiguration>() {
        @Override
        public DataSourceFactory getDataSourceFactory(MyApiConfiguration config) {
            return config.getDataSourceFactory();
        }
    });
}

📄 The Migration File (`migrations.xml`)

Place at src/main/resources/migrations.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">

    <!-- Each changeSet = one atomic database change -->
    <!-- id + author combination must be globally unique! -->

    <changeSet id="1" author="mitri">
        <createTable tableName="bookmarks">
            <column name="id" type="BIGINT" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="url" type="VARCHAR(500)">
                <constraints nullable="false"/>
            </column>
            <column name="title" type="VARCHAR(255)"/>
            <column name="tag" type="VARCHAR(100)"/>
        </createTable>
        <!-- Liquibase can auto-generate rollback for createTable (just DROP TABLE) -->
    </changeSet>

    <changeSet id="2" author="mitri">
        <addColumn tableName="bookmarks">
            <column name="created_at" type="TIMESTAMP" defaultValueDate="CURRENT_TIMESTAMP"/>
        </addColumn>
    </changeSet>

    <changeSet id="3" author="mitri">
        <!-- Insert test data — must specify manual rollback! -->
        <insert tableName="bookmarks">
            <column name="url" value="https://dropwizard.io"/>
            <column name="title" value="Dropwizard Official Site"/>
            <column name="tag" value="java"/>
        </insert>
        <rollback>
            <!-- Tell Liquibase how to undo this changeSet -->
            <delete tableName="bookmarks">
                <where>url = 'https://dropwizard.io'</where>
            </delete>
        </rollback>
    </changeSet>

</databaseChangeLog>

🚀 Running Migrations

# Check current state of your database
java -jar my-api.jar db status config.yml

# Apply all pending migrations
java -jar my-api.jar db migrate config.yml

# Apply migrations up to a specific tag
java -jar my-api.jar db migrate --include-tag mytag config.yml

# Roll back last 1 change
java -jar my-api.jar db rollback --count 1 config.yml

# Danger zone! Drops everything. Only for dev!
java -jar my-api.jar db drop-all --confirm-delete-everything config.yml

🆚 Liquibase vs Flyway

Feature	Liquibase	Flyway
Migration format	XML, YAML, SQL, JSON	SQL (primarily)
Rollbacks	Built-in support	Manual in Flyway Pro
Author tracking	Yes (prevents conflicts)	No
DB independence	Yes (XML format)	SQL-only = DB specific
Dropwizard default	✅ Yes	Via third-party bundle

Why Liquibase wins for teams: The author field prevents naming collisions. Two developers can't accidentally create conflicting migrations.

Chapter 9: Authentication

🔐 Dropwizard's Two Built-in Auth Methods

Basic Auth — Username + Password sent with every request
OAuth 2.0 — Token-based auth (used by most modern APIs)

📦 Dependency

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-auth</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

👤 Step 1 — The User Class

Must extend java.security.Principal in Dropwizard 1.x:

package com.example.auth;

import java.security.Principal;

public class User implements Principal {
    private final String name;

    public User(String name) {
        this.name = name;
    }

    @Override
    public String getName() {
        return name;
    }
}

🔑 Step 2 — The Authenticator

package com.example.auth;

import io.dropwizard.auth.AuthenticationException;
import io.dropwizard.auth.Authenticator;
import io.dropwizard.auth.basic.BasicCredentials;
import java.util.Optional;

public class MyAuthenticator implements Authenticator<BasicCredentials, User> {

    private final String validLogin;
    private final String validPassword;

    public MyAuthenticator(String login, String password) {
        this.validLogin = login;
        this.validPassword = password;
    }

    @Override
    public Optional<User> authenticate(BasicCredentials credentials)
            throws AuthenticationException {

        // Check if credentials match
        if (validLogin.equals(credentials.getUsername()) &&
            validPassword.equals(credentials.getPassword())) {
            return Optional.of(new User(credentials.getUsername()));
        }

        // Return empty Optional = authentication failed
        // Dropwizard will return 401 Unauthorized automatically
        return Optional.empty();
    }
}

🏭 Step 3 — Register in Application

@Override
public void run(MyApiConfiguration config, Environment environment) {

    // Create the authenticator with credentials from config
    MyAuthenticator authenticator = new MyAuthenticator(
        config.getLogin(),
        config.getPassword()
    );

    // Register Basic Auth with the authenticator
    environment.jersey().register(
        new AuthDynamicFeature(
            new BasicCredentialAuthFilter.Builder<User>()
                .setAuthenticator(authenticator)
                .setRealm("My Secure API")  // Shown in browser login dialog
                .buildAuthFilter()
        )
    );

    // Enable @Auth annotation injection in resources
    environment.jersey().register(new AuthValueFactoryProvider.Binder<>(User.class));
}

🔒 Step 4 — Protect Your Endpoints

@GET
@Path("/{id}")
@UnitOfWork
// @Auth injects the authenticated user AND protects the endpoint
public Bookmark getBookmark(@Auth User user,  // user = logged in user
                             @PathParam("id") long id) {
    // Only authenticated users reach here
    return dao.findById(id)
              .orElseThrow(() -> new NotFoundException("Not found: " + id));
}

📡 Making a Request with Basic Auth

# Using curl — -u "username:password"
curl -u "admin:secret123" http://localhost:8080/bookmarks/1

# With HTTPS (ignore self-signed cert for dev)
curl -k -u "admin:secret123" https://localhost:8443/bookmarks

# Or manually with header
curl -H "Authorization: Basic YWRtaW46c2VjcmV0MTIz" http://localhost:8080/bookmarks

Chapter 10: Health Checks & Metrics

💓 Health Checks

Health checks answer: "Is my app healthy right now?"

The admin port (8081) exposes:

/ping → returns "pong" if app is up
/healthcheck → runs all registered health checks

Built-in Health Checks

Dropwizard provides one by default:

DeadlockHealthCheck — checks if JVM has thread deadlocks

When you add the Hibernate bundle, you get a free extra:

Database connectivity check — tries to ping the database

Writing a Custom Health Check

package com.example.health;

import com.codahale.metrics.health.HealthCheck;

public class ExternalServiceHealthCheck extends HealthCheck {

    private final String serviceUrl;

    public ExternalServiceHealthCheck(String serviceUrl) {
        this.serviceUrl = serviceUrl;
    }

    @Override
    protected Result check() throws Exception {
        // Try to call the external service
        try {
            // ... make HTTP call to serviceUrl ...
            boolean isReachable = pingService(serviceUrl);

            if (isReachable) {
                return Result.healthy("External service is UP");
            } else {
                return Result.unhealthy("External service is DOWN at: " + serviceUrl);
            }
        } catch (Exception e) {
            return Result.unhealthy("Exception connecting: " + e.getMessage());
        }
    }

    private boolean pingService(String url) {
        // Simplified — use OkHttp (Chapter 15) in real code
        return true;
    }
}

Registering a Health Check

@Override
public void run(MyApiConfiguration config, Environment environment) {
    // Register your custom health check
    environment.healthChecks().register(
        "external-service",
        new ExternalServiceHealthCheck("https://api.thirdparty.com/ping")
    );
}

Response When Healthy

curl http://localhost:8081/healthcheck

{
  "deadlocks": { "healthy": true },
  "hibernate": { "healthy": true },
  "external-service": { "healthy": true, "message": "External service is UP" }
}

Response When Unhealthy (returns 500)

{
  "external-service": {
    "healthy": false,
    "message": "External service is DOWN at: https://api.thirdparty.com/ping"
  }
}

📊 Metrics

Dropwizard's Metrics library lets you measure things in your app:

import com.codahale.metrics.*;

public class BookmarkResource {
    private final Meter requests;      // How many requests per second
    private final Timer latency;       // How long requests take
    private final Counter activeUsers; // Count something
    private final Histogram payloadSize; // Distribution of values

    public BookmarkResource(MetricRegistry metrics, BookmarkDAO dao) {
        this.requests = metrics.meter("bookmark.requests");
        this.latency = metrics.timer("bookmark.latency");
        this.activeUsers = metrics.counter("bookmark.active-users");
    }

    @GET
    public List<Bookmark> getAllBookmarks() {
        requests.mark();  // Record a request happened
        try (Timer.Context ctx = latency.time()) {  // Measure how long this takes
            return dao.findAll();
        }
    }
}

📈 Sending Metrics to Graphite

# config.yml
metrics:
  reporters:
    - type: graphite
      host: graphite.example.com
      port: 2003
      prefix: myapp
      frequency: 1 minute

Chapter 11: Testing in Dropwizard

🧪 Testing Philosophy

Dropwizard takes testing seriously and provides special tools that make it easy to test REST APIs without running a full server.

📦 Dependency

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-testing</artifactId>
    <version>${dropwizard.version}</version>
    <scope>test</scope>
</dependency>
<!-- Note: JUnit is included automatically! No need to add it separately. -->

🧩 Level 1: Unit Testing a Resource (ResourceTestRule)

Tests just your resource class + Jersey, without a real database:

package com.example.resources;

import io.dropwizard.testing.junit.ResourceTestRule;
import org.junit.*;
import static org.mockito.Mockito.*;
import static org.assertj.core.api.Assertions.*;
import javax.ws.rs.core.Response;
import java.util.List;

public class BookmarkResourceTest {

    // Mock the DAO — no real database needed!
    private static final BookmarkDAO dao = mock(BookmarkDAO.class);

    // Spins up an in-memory Jersey server for testing
    @ClassRule
    public static final ResourceTestRule resources = ResourceTestRule.builder()
            .addResource(new BookmarkResource(dao))
            .build();

    @Before
    public void setup() {
        // Define what the mock returns
        when(dao.findAll()).thenReturn(List.of(new Bookmark(1L, "https://example.com", "Example")));
    }

    @After
    public void tearDown() {
        reset(dao);  // Reset mock between tests
    }

    @Test
    public void testGetAllBookmarks() {
        // Make a GET request using the test client
        List<Bookmark> result = resources.client()
                .target("/bookmarks")
                .request()
                .get(new GenericType<List<Bookmark>>(){});

        assertThat(result).hasSize(1);
        assertThat(result.get(0).getUrl()).isEqualTo("https://example.com");
        verify(dao).findAll();  // Verify DAO was called
    }

    @Test
    public void testCreateBookmark_returns201() {
        Bookmark newBookmark = new Bookmark(0, "https://new.com", "New Site");
        when(dao.save(any(Bookmark.class))).thenReturn(newBookmark);

        Response response = resources.client()
                .target("/bookmarks")
                .request()
                .post(Entity.json(newBookmark));

        assertThat(response.getStatus()).isEqualTo(201);
    }

    @Test
    public void testGetBookmark_notFound_returns404() {
        when(dao.findById(99L)).thenReturn(Optional.empty());

        Response response = resources.client()
                .target("/bookmarks/99")
                .request()
                .get();

        assertThat(response.getStatus()).isEqualTo(404);
    }
}

🔗 Level 2: Integration Testing (DropwizardAppRule)

Starts the entire application — tests everything together:

package com.example;

import io.dropwizard.testing.junit.DropwizardAppRule;
import org.junit.*;
import javax.ws.rs.client.*;
import javax.ws.rs.core.Response;

public class BookmarkIntegrationTest {

    // Starts the ENTIRE app (real server, real database)
    @ClassRule
    public static final DropwizardAppRule<MyApiConfiguration> RULE =
        new DropwizardAppRule<>(
            MyApiApplication.class,
            "config-test.yml"  // Use test config (H2 in-memory DB)
        );

    private Client client;

    @Before
    public void setup() {
        client = ClientBuilder.newClient();
    }

    @Test
    public void testServerIsRunning() {
        Response response = client
            .target(String.format("http://localhost:%d/bookmarks",
                RULE.getLocalPort()))
            .request()
            .get();

        assertThat(response.getStatus()).isEqualTo(200);
    }

    @Test
    public void testAuthenticatedEndpoint() {
        Response response = client
            .target(String.format("https://localhost:%d/bookmarks",
                RULE.getLocalPort()))
            .request()
            // Add Basic Auth header
            .header("Authorization", "Basic " + Base64.encode("admin:secret"))
            .get();

        assertThat(response.getStatus()).isEqualTo(200);
    }
}

🗃️ Testing Database Queries (DAOTest)

public class BookmarkDAOTest {

    // Spins up real Hibernate with H2 in-memory database
    @ClassRule
    public static final DAOTestRule database = DAOTestRule.newBuilder()
        .addEntityClass(Bookmark.class)
        .build();

    private BookmarkDAO dao;

    @Before
    public void setUp() throws Exception {
        dao = new BookmarkDAO(database.getSessionFactory());
    }

    @Test
    public void testSaveAndFind() {
        Bookmark saved = database.inTransaction(() -> {
            Bookmark b = new Bookmark();
            b.setUrl("https://test.com");
            b.setTitle("Test");
            return dao.save(b);
        });

        assertThat(saved.getId()).isGreaterThan(0);
        assertThat(dao.findById(saved.getId())).isPresent();
    }
}

Chapter 12: Lombok – Boilerplate Killer

😤 The Problem

Java is verbose. For every field in a class, you write:

A getter
A setter
Constructor(s)
equals() and hashCode()
toString()

That's 30+ lines of boring code for a simple 5-field class.

✨ Lombok's Solution

Annotations that generate all that code at compile time. You write the fields. Lombok writes the rest.

📦 Dependency

<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.30</version>
    <scope>provided</scope>  <!-- Only needed at compile time -->
</dependency>

Also install the Lombok plugin in IntelliJ IDEA (Settings → Plugins → search "Lombok").

🏷️ Core Lombok Annotations

import lombok.*;
import lombok.extern.slf4j.Slf4j;

// @Data = @Getter + @Setter + @ToString + @EqualsAndHashCode + @RequiredArgsConstructor
@Data
public class Bookmark {
    private long id;
    private String url;
    private String title;
}
// Lombok generates: getters, setters, toString(), equals(), hashCode(), constructor

Breakdown of each annotation:

@Getter              // Generates getters for all fields
@Setter              // Generates setters for all fields
@ToString            // Generates toString() like: Bookmark(id=1, url=https://...)
@EqualsAndHashCode   // Generates equals() and hashCode() based on fields
@NoArgsConstructor   // Generates: public Bookmark() {}
@AllArgsConstructor  // Generates: public Bookmark(long id, String url, String title) {}
@RequiredArgsConstructor  // Constructor for @NonNull and final fields only
public class Bookmark {
    private long id;
    private String url;
    private String title;
}

🏗️ @builder Pattern

Immutable objects with a fluent builder API:

@Builder
@Getter  // @Builder doesn't auto-generate getters
public class BookmarkRequest {
    private final String url;
    private final String title;
    private final String tag;
}

// Usage:
BookmarkRequest request = BookmarkRequest.builder()
    .url("https://example.com")
    .title("Example")
    .tag("java")
    .build();

🔒 Immutable Objects with @Value

// @Value = immutable version of @Data
// All fields are private final, only getters generated (no setters!)
@Value
public class UserId {
    long id;
    String username;
}

📝 Logging with @Slf4j

@Slf4j  // Generates: private static final Logger log = LoggerFactory.getLogger(BookmarkResource.class);
public class BookmarkResource {

    @GET
    public List<Bookmark> getAll() {
        log.info("Fetching all bookmarks");  // Use log directly!
        log.debug("Debug details here");
        log.error("Something went wrong", exception);
        return dao.findAll();
    }
}

⚠️ Lombok + Jackson Integration

Jackson needs a default constructor for deserialization. Use both:

@Data
@NoArgsConstructor          // Jackson needs this
@AllArgsConstructor         // Useful for your code
@JsonIgnoreProperties(ignoreUnknown = true)
public class Bookmark {
    private long id;
    private String url;
    private String title;
}

Or use @Builder with @Jacksonized:

@Builder
@Jacksonized  // Tells Jackson to use the builder for deserialization
@Value
public class Bookmark {
    long id;
    String url;
    String title;
}

Chapter 13: MapStruct – Object Mapping

🗺️ The Mapping Problem

In real applications, you have different object types:

Entity — maps to database (has Hibernate annotations)
DTO (Data Transfer Object) — what you send/receive in API
Domain Model — internal business logic object

You need to convert between them constantly. Manually writing conversion code is error-prone and tedious.

// Manual mapping — boring, error-prone
public BookmarkDTO toDTO(Bookmark entity) {
    BookmarkDTO dto = new BookmarkDTO();
    dto.setId(entity.getId());
    dto.setUrl(entity.getUrl());
    dto.setTitle(entity.getTitle());
    // Forget one field and you have a bug!
    return dto;
}

✨ MapStruct Solution

MapStruct generates these conversion methods at compile time using annotations.

📦 Dependency

<dependency>
    <groupId>org.mapstruct</groupId>
    <artifactId>mapstruct</artifactId>
    <version>1.5.5.Final</version>
</dependency>
<dependency>
    <groupId>org.mapstruct</groupId>
    <artifactId>mapstruct-processor</artifactId>
    <version>1.5.5.Final</version>
    <scope>provided</scope>
</dependency>

🔧 Maven Plugin (required with Lombok)

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <annotationProcessorPaths>
            <path>
                <groupId>org.projectlombok</groupId>
                <artifactId>lombok</artifactId>
                <version>1.18.30</version>
            </path>
            <path>
                <groupId>org.mapstruct</groupId>
                <artifactId>mapstruct-processor</artifactId>
                <version>1.5.5.Final</version>
            </path>
        </annotationProcessorPaths>
    </configuration>
</plugin>

📦 Define Your Objects

// Database Entity
@Entity
@Data
public class BookmarkEntity {
    @Id @GeneratedValue
    private Long id;
    private String url;
    private String title;
    private String tag;
    private Date createdAt;
    private String internalNotes;  // Should NOT be in API response!
}

// API DTO (Data Transfer Object)
@Data
@NoArgsConstructor
public class BookmarkDTO {
    private Long id;
    private String url;
    private String title;
    private String tag;
    // Notice: no internalNotes — we don't expose this!
    private String createdDate;  // Different format than entity!
}

🗺️ The Mapper Interface

package com.example.mapper;

import org.mapstruct.*;

@Mapper(componentModel = "default")  // Use "cdi" for CDI, "spring" for Spring
public interface BookmarkMapper {

    // Simple mapping — same field names map automatically!
    BookmarkDTO toDTO(BookmarkEntity entity);

    // Map from DTO back to entity
    BookmarkEntity toEntity(BookmarkDTO dto);

    // Custom field mapping
    @Mapping(source = "createdAt", target = "createdDate",
             dateFormat = "yyyy-MM-dd")  // Convert Date to formatted String
    @Mapping(target = "internalNotes", ignore = true)  // Don't map this field
    BookmarkDTO toDTOCustom(BookmarkEntity entity);

    // Map a list automatically!
    List<BookmarkDTO> toDTOList(List<BookmarkEntity> entities);
}

🏭 Getting the Mapper Instance

// Approach 1: Factory method (no DI)
BookmarkMapper mapper = Mappers.getMapper(BookmarkMapper.class);

// Approach 2: With Guice (Chapter 14)
// Bind in your module, inject where needed

✅ Using the Mapper in Resource

@Path("/bookmarks")
public class BookmarkResource {
    private final BookmarkDAO dao;
    private final BookmarkMapper mapper = Mappers.getMapper(BookmarkMapper.class);

    @GET
    @UnitOfWork
    public List<BookmarkDTO> getAllBookmarks() {
        List<BookmarkEntity> entities = dao.findAll();
        return mapper.toDTOList(entities);  // Entity → DTO, one line!
    }

    @POST
    @UnitOfWork
    public BookmarkDTO createBookmark(@Valid BookmarkDTO dto) {
        BookmarkEntity entity = mapper.toEntity(dto);  // DTO → Entity
        BookmarkEntity saved = dao.save(entity);
        return mapper.toDTO(saved);  // Entity → DTO for response
    }
}

Chapter 14: Google Guice – Dependency Injection

🤔 The Problem

As your app grows, you end up with this mess in your run() method:

public void run(Config config, Environment env) {
    HibernateBundle hib = hibernate;
    BookmarkDAO dao = new BookmarkDAO(hib.getSessionFactory());
    BookmarkMapper mapper = Mappers.getMapper(BookmarkMapper.class);
    MyAuthenticator auth = new MyAuthenticator(config.getLogin(), config.getPassword());
    BookmarkResource resource = new BookmarkResource(dao, mapper, auth);
    env.jersey().register(resource);
    // What if BookmarkResource needs 5 more dependencies?!
}

Every class manually creates its dependencies. Change one thing → change everywhere.

✨ Guice Solution

Inversion of Control: Instead of classes creating their dependencies, Guice creates them and injects them where needed.

📦 Dependency

<dependency>
    <groupId>com.google.inject</groupId>
    <artifactId>guice</artifactId>
    <version>5.1.0</version>
</dependency>
<!-- Dropwizard + Guice integration -->
<dependency>
    <groupId>ru.vyarus</groupId>
    <artifactId>dropwizard-guicey</artifactId>
    <version>5.7.1</version>
</dependency>

📝 Step 1 — Mark Dependencies with @Inject

// BookmarkDAO depends on SessionFactory
public class BookmarkDAO extends AbstractDAO<BookmarkEntity> {

    @Inject  // Guice will provide SessionFactory
    public BookmarkDAO(SessionFactory factory) {
        super(factory);
    }
}

// BookmarkResource depends on DAO and Mapper
public class BookmarkResource {

    private final BookmarkDAO dao;
    private final BookmarkMapper mapper;

    @Inject  // Guice will inject both!
    public BookmarkResource(BookmarkDAO dao, BookmarkMapper mapper) {
        this.dao = dao;
        this.mapper = mapper;
    }
}

📋 Step 2 — The Guice Module (Wiring Configuration)

package com.example;

import com.google.inject.AbstractModule;
import com.google.inject.Provides;
import com.google.inject.Singleton;

public class MyModule extends AbstractModule {

    @Override
    protected void configure() {
        // Bind interface → implementation
        // "Whenever someone asks for UserService, give them UserServiceImpl"
        bind(UserService.class).to(UserServiceImpl.class).in(Singleton.class);

        // Bind a class directly
        bind(BookmarkDAO.class).in(Singleton.class);

        // Bind mapper
        bind(BookmarkMapper.class).toInstance(Mappers.getMapper(BookmarkMapper.class));
    }

    // @Provides methods create instances that need configuration
    @Provides
    @Singleton
    public BookmarkDAO provideBookmarkDAO(SessionFactory factory) {
        return new BookmarkDAO(factory);
    }
}

🏭 Step 3 — Register with Dropwizard

Using dropwizard-guicey:

public class MyApiApplication extends GuiceApplication<MyApiConfiguration> {

    @Override
    public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
        super.initialize(bootstrap);
        bootstrap.addBundle(hibernate);
    }

    @Override
    public void run(MyApiConfiguration config, Environment environment) {
        // Resources are auto-discovered and injected!
    }

    @Override
    protected Injector createInjector(MyApiConfiguration config) {
        return Guice.createInjector(new MyModule());
    }
}

🏷️ Key Guice Annotations

@Inject     // Mark constructor/field/method for injection
@Singleton  // One instance for entire app lifetime
@Named("dbUrl")  // Disambiguate when multiple bindings of same type exist

// Usage:
@Inject @Named("dbUrl") String databaseUrl;

🔄 Guice Scopes

bind(BookmarkDAO.class).in(Singleton.class);    // One instance per app
bind(BookmarkDAO.class).in(RequestScoped.class); // New instance per HTTP request
bind(BookmarkDAO.class);                         // Default: new instance every time

Chapter 15: OkHttp – HTTP Client

🌐 Why OkHttp?

Microservices talk to each other over HTTP. You need an HTTP client to:

Call third-party APIs (payment, email, maps)
Call other microservices in your system

OkHttp is the most popular Java HTTP client — fast, simple, feature-rich.

📦 Dependency

<dependency>
    <groupId>com.squareup.okhttp3</groupId>
    <artifactId>okhttp</artifactId>
    <version>4.11.0</version>
</dependency>

🚀 Basic GET Request

import okhttp3.*;
import java.io.IOException;

public class ExternalApiClient {

    // IMPORTANT: Reuse the same OkHttpClient — it manages connection pooling!
    private final OkHttpClient client = new OkHttpClient();
    private final ObjectMapper objectMapper = new ObjectMapper();

    public String fetchData(String url) throws IOException {
        // Build the request
        Request request = new Request.Builder()
            .url(url)
            .header("Accept", "application/json")
            .header("Authorization", "Bearer my-token")
            .build();

        // Execute synchronously
        try (Response response = client.newCall(request).execute()) {
            if (!response.isSuccessful()) {
                throw new IOException("Unexpected response: " + response);
            }
            return response.body().string();
        }
    }
}

📤 POST Request with JSON Body

public static final MediaType JSON = MediaType.get("application/json; charset=utf-8");

public Bookmark createRemoteBookmark(BookmarkDTO dto) throws IOException {
    String json = objectMapper.writeValueAsString(dto);

    Request request = new Request.Builder()
        .url("https://api.other-service.com/bookmarks")
        .post(RequestBody.create(json, JSON))  // POST with JSON body
        .build();

    try (Response response = client.newCall(request).execute()) {
        String responseBody = response.body().string();
        return objectMapper.readValue(responseBody, Bookmark.class);
    }
}

⏱️ Timeouts & Configuration

OkHttpClient client = new OkHttpClient.Builder()
    .connectTimeout(10, TimeUnit.SECONDS)  // How long to wait to connect
    .readTimeout(30, TimeUnit.SECONDS)     // How long to wait for data
    .writeTimeout(15, TimeUnit.SECONDS)    // How long to wait for upload
    .retryOnConnectionFailure(true)        // Retry on network errors
    .build();

🔄 Async Requests

public void fetchDataAsync(String url, Callback callback) {
    Request request = new Request.Builder().url(url).build();

    // Non-blocking — callback is called when done
    client.newCall(request).enqueue(new Callback() {
        @Override
        public void onFailure(Call call, IOException e) {
            log.error("Request failed", e);
        }

        @Override
        public void onResponse(Call call, Response response) throws IOException {
            try (response) {
                String body = response.body().string();
                // Process response
            }
        }
    });
}

🔌 OkHttp Interceptors (Middleware)

Interceptors run for every request — perfect for logging, auth headers, etc.:

// Logging interceptor
OkHttpClient client = new OkHttpClient.Builder()
    .addInterceptor(chain -> {
        Request request = chain.request();
        log.info("→ {} {}", request.method(), request.url());

        long start = System.currentTimeMillis();
        Response response = chain.proceed(request);
        long duration = System.currentTimeMillis() - start;

        log.info("← {} {} ({}ms)", response.code(), request.url(), duration);
        return response;
    })
    .addInterceptor(chain -> {
        // Auth interceptor — add token to every request
        Request authenticated = chain.request().newBuilder()
            .header("Authorization", "Bearer " + getToken())
            .build();
        return chain.proceed(authenticated);
    })
    .build();

Chapter 16: Hystrix – Circuit Breaker

⚡ The Cascading Failure Problem

Service A calls Service B. Service B calls Service C. Service C goes down.

Without circuit breakers:

Service C times out after 30 seconds
Service B waits 30s, then times out
Service A waits 30s too = everything hangs for 30+ seconds!
Threads pile up, memory fills up, your whole system crashes

🔌 The Circuit Breaker Pattern

Like an electrical circuit breaker — when failures exceed a threshold, it "trips" and subsequent calls fail immediately instead of waiting.

States:

CLOSED — Normal operation, calls pass through
OPEN — Too many failures, all calls fail immediately
HALF-OPEN — Testing if service recovered; allows one call through

📦 Dependency

<dependency>
    <groupId>com.netflix.hystrix</groupId>
    <artifactId>hystrix-core</artifactId>
    <version>1.5.18</version>
</dependency>

☕ Wrapping Calls with HystrixCommand

package com.example.hystrix;

import com.netflix.hystrix.*;
import java.io.IOException;

public class FetchBookmarkCommand extends HystrixCommand<String> {

    private final String bookmarkId;
    private final ExternalApiClient apiClient;

    public FetchBookmarkCommand(String bookmarkId, ExternalApiClient apiClient) {
        super(HystrixCommandGroupKey.Factory.asKey("BookmarkService"));
        this.bookmarkId = bookmarkId;
        this.apiClient = apiClient;
    }

    @Override
    protected String run() throws Exception {
        // The actual (potentially failing) call
        return apiClient.fetchData("https://api.service.com/bookmarks/" + bookmarkId);
    }

    @Override
    protected String getFallback() {
        // This runs when:
        // 1. run() throws an exception
        // 2. Circuit is OPEN (previous failures)
        // 3. Timeout occurs
        return "{\"id\":\"" + bookmarkId + "\",\"status\":\"unavailable\"}";
    }
}

🎯 Using the Command

// Synchronous execution
String result = new FetchBookmarkCommand(bookmarkId, client).execute();

// Asynchronous (returns Future)
Future<String> future = new FetchBookmarkCommand(bookmarkId, client).queue();

// Reactive (returns Observable)
Observable<String> observable = new FetchBookmarkCommand(bookmarkId, client).observe();

⚙️ Configuring Hystrix

HystrixCommand.Setter config = HystrixCommand.Setter
    .withGroupKey(HystrixCommandGroupKey.Factory.asKey("BookmarkService"))
    .andCommandKey(HystrixCommandKey.Factory.asKey("FetchBookmark"))
    .andCommandPropertiesDefaults(
        HystrixCommandProperties.Setter()
            .withExecutionTimeoutInMilliseconds(3000)  // 3 second timeout
            .withCircuitBreakerRequestVolumeThreshold(10)  // Min requests before circuit can trip
            .withCircuitBreakerErrorThresholdPercentage(50)  // Trip if 50% fail
            .withCircuitBreakerSleepWindowInMilliseconds(5000)  // Wait 5s before trying again
    );

// Use in command:
public FetchBookmarkCommand(String id, ExternalApiClient client) {
    super(config);  // Pass config to super
    this.bookmarkId = id;
    this.apiClient = client;
}

🏎️ HystrixCommand + OkHttp (Combined Example)

@Slf4j
public class ExternalServiceClient {

    private final OkHttpClient httpClient;
    private final ObjectMapper objectMapper;

    @Inject
    public ExternalServiceClient(OkHttpClient httpClient, ObjectMapper objectMapper) {
        this.httpClient = httpClient;
        this.objectMapper = objectMapper;
    }

    public BookmarkDTO getBookmark(String id) {
        // Wrap the OkHttp call in Hystrix
        return new HystrixCommand<BookmarkDTO>(
            HystrixCommandGroupKey.Factory.asKey("ExternalBookmarks")) {

            @Override
            protected BookmarkDTO run() throws IOException {
                Request request = new Request.Builder()
                    .url("https://external.api.com/bookmarks/" + id)
                    .build();

                try (Response response = httpClient.newCall(request).execute()) {
                    return objectMapper.readValue(response.body().string(), BookmarkDTO.class);
                }
            }

            @Override
            protected BookmarkDTO getFallback() {
                log.warn("Fallback triggered for bookmark: {}", id);
                BookmarkDTO fallback = new BookmarkDTO();
                fallback.setTitle("Unavailable");
                return fallback;
            }
        }.execute();
    }
}

Chapter 17: 🔲 TODO — Advanced Topics to Study Next

These topics weren't covered in this guide but are important for production-ready applications:

🔐 Security

[ ] OAuth 2.0 — Token-based authentication (JWT tokens)
[ ] HTTPS/TLS — Configure SSL in Dropwizard config.yml
[ ] LDAP Authentication — Enterprise authentication via directory services
[ ] CORS — Cross-Origin Resource Sharing configuration

📊 Observability

[ ] Distributed Tracing — Zipkin/Jaeger for tracing requests across microservices
[ ] Prometheus — Modern metrics collection (alternative to Graphite)
[ ] Grafana Dashboards — Visualizing your metrics
[ ] Structured Logging — JSON logs for log aggregation (ELK Stack)
[ ] OpenTelemetry — Unified observability standard

🏗️ Architecture

[ ] Authorization — @RolesAllowed, @PermitAll annotations
[ ] Rate Limiting — Prevent API abuse
[ ] API Versioning — Strategies for versioning your REST API
[ ] HATEOAS — Hypermedia-driven REST (links in responses)
[ ] GraphQL — Alternative to REST for flexible queries
[ ] OpenAPI/Swagger — Auto-generate API documentation

🗄️ Database

[ ] JDBI — Lighter-weight alternative to Hibernate
[ ] Connection Pooling — HikariCP configuration
[ ] Read Replicas — Routing reads vs writes
[ ] Database Sharding — Horizontal scaling

🐳 Deployment

[ ] Docker — Containerizing your Fat JAR
[ ] Kubernetes — Orchestrating containers
[ ] CI/CD — GitHub Actions / Jenkins pipelines
[ ] Blue/Green Deployment — Zero-downtime deployments
[ ] Service Discovery — Consul, Eureka

⚡ Performance

[ ] Async Resources — @Suspended and AsyncResponse in JAX-RS
[ ] Caching — Redis/Memcached with Dropwizard
[ ] Connection Timeout Tuning — Jetty thread pool settings
[ ] GraalVM Native Image — Compile to native binary for faster startup

🧪 Advanced Testing

[ ] Consumer-Driven Contract Testing — Pact framework
[ ] WireMock — Mock external HTTP services in tests
[ ] TestContainers — Real databases in Docker for integration tests
[ ] Chaos Engineering — Testing failure scenarios (Chaos Monkey)

📨 Messaging

[ ] Apache Kafka — Event streaming
[ ] RabbitMQ — Message queuing
[ ] Dropwizard Kafka Bundle — Integration

DEV Community: Ajinkya Singh

Pull, Don't Push: Fixing Load Imbalance with a Shared Task Queue

Recap

The actual problem with fixed slicing

The real implementation: tq::Master and tq::Worker

The shared cursor

The worker loop

Wiring it up

Why the atomic cursor specifically, not a mutex-protected queue

Results — same worst-case dataset, 100 chunks

Takeaways

Load Balancing in C++ Multithreading: Why Equal Splits Aren't Equal Work

Recap

What we're actually studying here: unbalanced load

The task: not all work is the same size

Three ways to lay out the same tasks

Measuring the imbalance, not just guessing at it

Why this is invisible if you only look at total time

Results — 100 chunks, each layout, same machine

Takeaways

False Sharing, Measured: Two Threads, Zero Shared Variables, Still Slow

Recap

The setup: separate variables, same benchmark

The simple fix already sitting in this series' code: alignas(64)

The controlled experiment: prac::FalseSharing

Results — run back to back, same machine

Why the exact same fix helped in one case and did nothing in the other

Why this is the strangest bug in the series so far

Takeaways

Multi-Core Cache Coherency: The Invisible Lock You Never Wrote

Recap — and the question this article finally answers

The new problem: private caches, shared memory

The four states every cached line is tagged with (MESI)

What actually crosses the wires

A cost ladder — not every core-to-core interaction is equal

The key insight: invalidation happens per line, not per variable

Wait — doesn't this mean coherency already prevents races?

Glossary added this part

Takeaways

The 64-Byte Truck: Cache Lines Explained

Recap

What actually happens when you read one int

Why 64 bytes specifically — two independent reasons

Reason 1: the physical width of the connection

Reason 2: spatial locality says it's worth it

Cache lines are aligned, not centered on what you asked for

How much fits in one line

Glossary added this part

Takeaways

The Hardware Memory Hierarchy: From L1 Caches to Main RAM

Recap

The problem caches solve

The hierarchy, level by level

Why a pyramid, not one big fast memory

How the hierarchy is actually used

Why caching works at all: locality

Glossary added this part

Takeaways

How CPUs and Memory Actually Work: Bytes, Addresses, and Latency

Recap — the question this series has been avoiding

A computer, stripped down to two things talking to each other

Cycles — the CPU's metronome

Bytes and addresses — a variable is just a name for a location

Fast vs. slow — the ordering that everything else hangs on

Cores — why "more than one CPU" changes everything

Takeaways

C++ Multithreading: Why -O2 Deletes Your Loops and How std::atomic Fixes It

Recap

The setup: a flag, and a thread that waits on it

The problem: the compiler doesn't just get this wrong — it gets it wrong differently at every optimization level

Why: the compiler is allowed to assume this bug doesn't exist

std::atomic actually does two different jobs — don't blur them together

Back to ready — this article only ever needed Job 2

The fix: std::atomic<bool>

Why this matters more than the Part 1 bug did

Takeaways

Understanding C++ Threads: Data Races Explained Down to the Assembly

Notes & References

The setup

The problem: the obvious multithreaded version is wrong

The real implementation: `tq::Master` and `tq::Worker`

The simple fix already sitting in this series' code: `alignas(64)`

The controlled experiment: `prac::FalseSharing`

What actually happens when you read one `int`

`std::atomic` actually does two different jobs — don't blur them together

Back to `ready` — this article only ever needed Job 2

The fix: `std::atomic<bool>`

Why: `cnt += x` is not one instruction