DEV Community: Mukul S

Stop Paying for the Same Tokens Twice: A Practical Guide to Prompt Caching

Mukul S — Wed, 05 Aug 2026 19:03:27 +0000

You've built a chatbot. Every turn, you re-send the whole conversation — the 8,000-token system prompt, the uploaded PDF, the 15 messages of history — just so the model can answer "and what about Mars?"
The model re-reads all of it. Every. Single. Time. You pay full price for all of it. Every. Single. Time.
Prompt caching fixes this. It's roughly one extra line of JSON, and it can cut your input costs by ~90% on the repeated part while making responses noticeably faster.

Let's walk through it.

The one-liner version

Add cache_control at the top level of your request:

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},   # <-- this is the whole trick
    system="You are an AI assistant tasked with analyzing literary works...",
    messages=[
        {"role": "user", "content": "Analyze the major themes in 'Pride and Prejudice'."}
    ],
)
print(response.usage)

That's automatic caching. The API caches everything up to and including the last cacheable block in your request. Next time you send a request that starts with the same content, that prefix is read from cache instead of reprocessed.
Same thing in curl, if that's more your speed:

curl https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-opus-5",
    "max_tokens": 1024,
    "cache_control": {"type": "ephemeral"},
    "system": "You are an AI assistant tasked with analyzing literary works.",
    "messages": [{"role": "user", "content": "Analyze the major themes in Pride and Prejudice."}]
  }'

The mental model: it's a prefix cache

This is the single most important thing to internalize.
Your prompt is read in a fixed order:

tools  →  system  →  messages

Caching works on prefixes of that sequence. When you mark a block with cache_control, you're saying: "cache everything from the start of the request up to and including this block."
Which means:

✅ Static stuff at the front = cacheable.
❌ Change something early = everything after it is invalidated. So the golden rule is: put the boring, unchanging stuff first. Tool definitions, system instructions, that 50-page contract, your 20 few-shot examples. Then the volatile stuff — the user's actual question — goes last. ---

What it costs (and saves)

Three price tiers instead of one:

	Multiplier vs. base input
5-minute cache write	1.25×
1-hour cache write	2×
Cache read (hit)	0.1×

So for Claude Opus 5 ($5/MTok input):

First request writes the cache: $6.25/MTok
Every subsequent hit: $0.50/MTok You pay a 25% premium once, then 90% off forever after. If you reuse a prefix even twice, you're already ahead. Cache breakpoints themselves are free. You're never charged for having a breakpoint — only for tokens actually written and read.

Reading the usage fields (this trips everyone up)

"usage": {
  "cache_read_input_tokens": 100000,
  "cache_creation_input_tokens": 0,
  "input_tokens": 50
}

input_tokens is not your total input. It's only the tokens after your last cache breakpoint. Total is:

total = cache_read_input_tokens + cache_creation_input_tokens + input_tokens

So the example above processed 100,050 tokens, not 50. Handy side effect: cache hits don't count against your rate limits the way fresh input does, so effective throughput goes up too.

Quick sanity check: if both cache_creation_input_tokens and cache_read_input_tokens are 0, nothing was cached. Most likely you're under the minimum length (see below) — the API won't error, it just silently skips caching.

Minimum sizes — don't skip this

Prompts shorter than a per-model floor simply won't cache:

Model	Minimum cacheable tokens
Opus 5, Fable 5, Mythos 5	512
Opus 4.8, Sonnet 5 / 4.6 / 4.5	1,024
Opus 4.7, Haiku 3.5	2,048
Opus 4.6, Opus 4.5, Haiku 4.5	4,096

If you're just under the line, it's often worth padding the cached section (more examples, more context) to get over it. Cache reads are cheap enough that the extra tokens pay for themselves.

Multi-turn conversations: let it drive

With automatic caching, the breakpoint walks forward on its own as the conversation grows:

Request	What happens
1	`System + U1 + A1 + U2` → everything written to cache
2	`... + A2 + U3` → System→U2 read from cache, A2+U3 written
3	`... + A3 + U4` → System→U3 read from cache, A3+U4 written

No bookkeeping. No moving markers around. Each turn reads the whole prior conversation from cache and only writes the new bit. This is the single best default for chat apps.

Explicit breakpoints: when you need the wheel

Put cache_control on individual blocks when different parts of your prompt change at different rates. You get up to 4 breakpoints.
Classic RAG-agent layout:

{
  "tools": [ /* ... */, { "name": "get_document", "cache_control": {"type": "ephemeral"} } ],
  "system": [
    { "type": "text", "text": "You are a research assistant...", "cache_control": {"type": "ephemeral"} },
    { "type": "text", "text": "# Knowledge Base\n## Doc 1...", "cache_control": {"type": "ephemeral"} }
  ],
  "messages": [
    { "role": "user", "content": [
      { "type": "text", "text": "Tell me about Perseverance.", "cache_control": {"type": "ephemeral"} }
    ]}
  ]
}

Four independent segments:

Tools — basically never change
Instructions — change on deploys
RAG documents — change daily
Conversation — changes every turn Swap the RAG docs and you keep segments 1 and 2. Add a turn and you keep 1, 2, and 3. Changes only invalidate their own segment and everything downstream.

The mistake literally everyone makes

Here's the bug I want you to remember, because it's expensive and silent.
Your prompt: blocks 1–5 are a big static system context. Block 6 is f"[{timestamp}] {user_message}". You put cache_control on block 6, because it's the end and that seems right.

Request 1: cache written at block 6. The hash includes the timestamp.
Request 2: different timestamp → different hash → miss. The system walks back through blocks 5, 4, 3, 2, 1 looking for entries... but no request ever wrote an entry there.
Result: a fresh cache write every single request. You pay the 1.25× premium forever and never get a single read. The lookback does not find stable content behind your breakpoint and cache it for you. It only finds entries that earlier requests wrote — and writes happen only at breakpoints. The fix is one line: move cache_control to block 5, the last block that's identical across requests. > Rule of thumb: put the breakpoint on the last block whose prefix is identical across the requests you want to share a cache. (Note: automatic caching falls into the same trap here, since it targets the last block. If your final block has a per-request timestamp, use an explicit breakpoint on the static prefix instead.)

The 20-block lookback window

A related gotcha. When looking for a cache hit, the system checks your breakpoint's position and then walks backward — but only 20 blocks.

Turn 1: 10 blocks, breakpoint at 10. Entry written at 10.
Turn 2: 15 blocks, breakpoint at 15. Walks back to 10, finds turn 1's entry. Hit! Only blocks 11–15 processed fresh.
Turn 3: 35 blocks, breakpoint at 35. Checks blocks 35 down to 16, finds nothing. The turn-2 entry at block 15 is one position outside the window. Miss. Full reprocess. If your conversation can jump by 20+ blocks in a single turn, add a second breakpoint further back so a write accumulates there before you need it.

The 5-minute vs 1-hour decision

Default TTL is 5 minutes, and it refreshes for free every time you hit the cache. An active chat session basically keeps itself warm.

"cache_control": { "type": "ephemeral", "ttl": "1h" }

Reach for 1h when:

Follow-ups are likely to land after 5 minutes but within an hour (a user who steps away; an agent sub-task that runs long)
Latency matters on those delayed follow-ups
You're batching, where jobs commonly take 5–60 minutes Stick with 5m when your prompt is used more often than every 5 minutes — refreshes are free, so you'd be paying 2× writes for nothing. Mixing TTLs in one request is allowed, with one rule: longer TTLs must come first. 1-hour blocks before 5-minute blocks.

Bonus: pre-warm the cache

Latency-sensitive app? The first user of the day eats the cache-miss penalty. Unless you warm it up first:

SYSTEM_PROMPT = [{
    "type": "text",
    "text": "You are an expert software engineer...",
    "cache_control": {"type": "ephemeral"},
}]
def prewarm_cache():
    client.messages.create(
        model="claude-opus-5",
        max_tokens=0,                                  # no output generated
        system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": "warmup"}],
    )

max_tokens: 0 reads your prompt in, writes the cache at your breakpoint, and returns immediately with an empty content array and stop_reason: "max_tokens". Zero output tokens billed. (You still pay the cache write, naturally.)
Two things to get right:

Put the breakpoint on the shared content (your system prompt), not on the "warmup" placeholder — otherwise the entry is keyed to the placeholder and real traffic never hits it. This is why pre-warming needs an explicit breakpoint rather than automatic caching.
Use the same thinking config and effort setting as your real requests. Those get rendered into the prompt, so a mismatched pre-warm writes an entry nobody uses. max_tokens: 0 is rejected with stream: true, extended thinking, structured outputs, forced tool_choice, or inside a Batches request.

What breaks the cache

Cache hits need a 100% byte-identical prefix. Things that invalidate:

Change	Blast radius
Tool definitions	Everything
Toggling web search / citations	System + messages
Switching fast mode	System + messages
`tool_choice`	Messages
Adding/removing images anywhere	Messages
Thinking config or `effort`	Messages (and sometimes more)

One sneaky one: some languages (Go, Swift) randomize map key order when serializing JSON. If your tool_use blocks come out with shuffled keys, your cache never hits and you'll have no idea why. Pin the ordering.
Also: caches are isolated per organization, and per workspace on the Claude API. And a cache entry only becomes available after the first response begins — so firing 10 parallel requests with the same prefix gives you 10 misses. Send one, wait, then fan out.

Troubleshooting checklist

Cache not hitting? Run down this list:

[ ] Is the cached section byte-identical across calls?
[ ] Are you over the minimum token count for your model?
[ ] Is the breakpoint on a block that stays the same (no timestamps, no user input)?
[ ] Are calls landing within the TTL?
[ ] Are tool_choice, image presence, thinking config, and effort consistent?
[ ] Is your JSON key order stable?
[ ] Has a growing conversation pushed you past the 20-block lookback?

Got a caching setup that surprised you — good or bad? Drop it in the comments. 🚀

Skills as Sub-Agents: Orchestrating Complex work with Claude Skills

Mukul S — Fri, 31 Jul 2026 17:45:24 +0000

If you've built anything with a coding agent, you've hit the wall: the task is too big for one prompt. You ask it to "find out what's causing this bug," and it starts strong — then drowns. Half its context is the raw output of files it dumped, it's lost the thread of which theory it was testing, and the answer it finally gives is confidently wrong.
The problem isn't the model. It's that you asked one agent, with one context window, to do everything — gather evidence, hold it all in its head, and reason about it at the same time.
There's a cleaner pattern. Split the work into skills, and have one skill orchestrate the others as sub-agents. Let's walk through it.

New to skills or the agent loop? The one-line version: a skill is a packaged set of instructions on disk that the agent loads only when it's relevant. A sub-agent is a fresh agent instance with its own separate context window. This post is about combining the two.

The core idea: reasoning up top, work down below

Picture two layers.
The orchestrator sits on top. Its only job is to think: plan the investigation, interpret results, decide what to do next, and conclude. It never opens a file, never greps the codebase, never parses output. The first lines of an orchestrator skill should say exactly that:

You are a reasoning and coordination layer. You think, you plan, you interpret, you propose — but you never dig into the data yourself. That work belongs in sub-agents.
The workers sit below. Each is a small, focused skill that does one concrete thing — find every usage of a function, read a file's git history, run a specific test. The orchestrator spawns each worker as a sub-agent: a separate agent with its own fresh context window.

Why does this split matter so much? Context hygiene.
When a worker greps a huge codebase and reads a dozen files, all of that lands in the worker's context — not the orchestrator's. The worker chews through it, extracts the one thing that matters, and reports back a three-line summary. The orchestrator's context stays clean: it accumulates conclusions, not raw data. That's the whole trick, and it's why the orchestrator can run for dozens of steps without falling over.

Anatomy of a worker skill

A worker skill is just a folder with a SKILL.md file. The front matter does the heavy lifting:

---
name: find-usages
description: "Find every place a symbol, function, or pattern is used across the codebase. Use when tracing how a change ripples through the code."
context: fork
allowed-tools: Grep, Read
argument-hint: <symbol> [--path=<dir>]
---
# Find Usages
## Input
- **Symbol** (required): the function, class, or pattern to trace.
- **Path** (optional): limit the search to a directory.
- **Output file**: write your full report here.
## Procedure
1. Search the codebase for the symbol.
2. For each hit, read enough surrounding code to classify it
   (definition, call site, test, re-export).
3. Write the full annotated list to the output file.
4. Return a 3-line summary: total count, and the 1–2 most relevant sites.

Four things are worth calling out, because they're what make it composable:

description — this is the skill's résumé. It's how the orchestrator decides whether to call this worker at all (more on that below). Write it for selection, not just documentation.
context: fork — run this skill in an isolated context. This is what makes it a sub-agent instead of inline instructions.
allowed-tools — scope the worker down to only what it needs. A read-only search worker gets Grep, Read and nothing that can modify files. This is your safety boundary, enforced per-skill.
The output contract — the worker writes its full findings to a file and returns only a summary. This is the single most important convention in the whole system. Get it right and everything else falls into place.

The summary-vs-report split

Say it plainly, because it's the load-bearing idea:

A sub-agent writes its full report to a file, and returns only a short summary to the orchestrator.

The orchestrator reads the summary immediately. If it needs the details — the exact line, the full list — it reads the file on demand. Most of the time it doesn't need to. So the expensive raw data lives on disk, referenced by path, instead of clogging the reasoning layer's context.

How the orchestrator picks workers (without reading them all)

Here's a subtlety that trips people up. If the orchestrator has twenty worker skills available, you might think it needs to read all twenty SKILL.md files to know what they do. That would blow its context before the work even starts.
It doesn't. The agent already sees every skill's name and one-line description — that's the progressive-disclosure model skills are built on. So the orchestrator picks workers by description alone. Make this an explicit rule in the orchestrator:

Do NOT read the full worker SKILL.md files — they are large and will pollute your context. The skill name and description are enough to pick.
This is why that description field is so important. It's not documentation for humans; it's the interface the orchestrator selects against.

The orchestration loop

Put it together and the orchestrator runs a loop that should feel familiar if you've seen the basic agent loop — just one level up. Underneath any specific task it's the same four beats:

PLAN (once)
  Confirm the request, scope it, restate the goal.
repeat:
  DISPATCH    → spawn worker sub-agents in parallel to do the next chunk
  INTEGRATE   → read each summary as it returns, update your working picture
  DECIDE      → goal met? finish. gaps remain? dispatch more.
DELIVER
  Present the result (with alternatives called out if you're unsure) and stop.

That's the general shape. Each kind of task just fills in the blanks differently:

Task type	DISPATCH gathers…	The "working picture" is…	DELIVER produces…
Debugging	evidence about a failure	competing hypotheses	root cause + fix
Research	facts from sources	an outline of the answer	a synthesized report
Migration	which files need changing	a checklist of edits	a completed, verified change
Audit	findings per rule/area	a running list of issues	a prioritized report

The vocabulary changes; the loop doesn't. Whenever a task is "break it into independent chunks, do them, combine the results, repeat until done," this is the pattern.
A few things make it work well in practice:
Fan out, then integrate as results arrive. The orchestrator spawns workers as background sub-agents and doesn't block waiting for all of them. As each summary comes back, it folds it into the picture and — crucially — can dispatch a follow-up immediately. A usage search turns up a suspicious call site? Fire off a git-history worker on that file now, while the other workers are still running. The loop is interleaved, not batch-then-wait.
Cap the expensive workers. Cheap, read-only workers (a quick search, a file read) have no concurrency limit — the cost of running one you didn't need is near zero. Heavier workers (running the full test suite, deep analysis) get a hard cap, e.g. "at most 2 at a time." Match the limit to the cost.
Refresh the instructions each phase. On a long run, the orchestrator's own SKILL.md scrolls out of its context. So each phase's detailed rules live in a separate file (phase-collect.md, phase-synthesize.md…) that the orchestrator re-reads every time it enters that phase. Don't rely on the agent remembering a rule it read forty steps ago — put the rule back in front of it when it's about to act on it.
Keep an append-only log. One file is the run's source of truth: every dispatch, every result, every decision. Append-only — if an earlier note was wrong, you add a correction rather than editing history. It's the audit trail and the memory that survives context churn.

Why this beats one big prompt

Context stays clean. Raw data is quarantined in worker contexts. The reasoning layer only ever sees distilled conclusions, so it can go deep and long without degrading.
Real parallelism. Ten independent checks run at once instead of one after another. A job that would take an agent twenty sequential steps finishes in a handful of rounds.
Specialization. Each worker is small, testable, and good at exactly one thing. You can improve the usage-finder without touching anything else.
Safety per worker. allowed-tools means your read-only workers cannot modify files, no matter what the model decides. The boundary is structural, not a polite request in a prompt.
Composability. Adding a capability means adding a skill folder. The orchestrator picks it up by description — no changes to the loop.

Setting up your own

The pattern transfers to any complex task — debugging, research, migrations, audits. A starting recipe:

Write the workers first. For each concrete sub-task, make a skill folder with a SKILL.md. Nail the description (for selection), set context: fork, scope allowed-tools, and define the input params + output-file contract.
Enforce summary-vs-report everywhere. Every worker writes full output to a file, returns a short summary. Non-negotiable — it's what keeps the orchestrator's context alive.
Write the orchestrator as pure reasoning. Tell it, in the strongest terms, to delegate all the actual work. Give it a loop (gather → interpret → decide) and the rule to pick workers by description, not by reading them.
Externalize per-phase rules. Put detailed checklists in phase files the orchestrator re-reads on entry, so they stay in its attention on long runs.
Cap the expensive workers and log everything. Concurrency limits by cost; an append-only log as the source of truth. That's the architecture. One agent that thinks, many that fetch — each in its own context, each doing one job well. Once you stop cramming everything into a single prompt and start treating skills as sub-agents, the ceiling on what an agent can reliably handle goes way, way up.

How Claude Code Actually Works: The Agentic Loop

Mukul S — Wed, 29 Jul 2026 22:26:41 +0000

Coding agents feel like magic. They aren't. Here's the tiny loop at the heart of it -- the one you could sketch on a napkin.

You type "fix the failing test." Claude Code reads a file, runs pytest, sees a red failure, edits the code, runs pytest again, sees green, and tells you it's done.

The first time you watch this happen, it feels like magic.

It isn't. Underneath, there's a loop so small you could sketch it on a napkin. Once you see it, coding agents stop being mysterious -- and you'll understand why they behave the way they do.

Let's build the mental model from scratch.Here's the unlock that makes everything else click:

A language model can't do everything. It can only produce text.

The model can't open your files. It can't run a command. It can't touch the network. Drop it on a desert island and it will happily generate paragraphs forever, but it will never actually run your tests.

So how does Claude Code edit files and run commands?

It doesn't. A regular program does -- and the model just tells it what to do.

Enter the harness

Around the model sits an ordinary program. Call it the harness (that's the CLI you're running).

The deal between them is simple:

The harness tells the model, up front: "Here are the tools you can use."
The model replies with a tool call -- not prose, but a structured request like "run pytest -q".
The harness actually runs it, captures the output, and hands the result back to the model.
The model reads that result and decides what to do next.

The model is the brain. The harness is the hands. Neither works without the other.

Tools are just a menu + a contract

When a session starts, the harness gives the model a menu of tools, each with a name, a description, and a schema for its inputs. Simplified, it looks like this:

{
  "name": "run_bash",
  "description": "Execute a shell command and return its output",
  "input_schema": {
    "command": "string"
  }
}

Now, when the model wants to run the tests, it doesn't say "you should run pytest." It emits a structured tool call:

{
  "tool": "run_bash",
  "input": { "command": "pytest -q" }
}

The harness executes that command and appends the result to the conversation:

{
  "tool_result": "1 failed, 4 passed\nE   assert add(2, 2) == 5"
}

That result goes back into the model on the next turn. Now the model can react -- it knows the test failed and exactly why.

This back-and-forth is the whole game.

The loop

Here's the engine. This is the napkin sketch:

conversation = [system_prompt, user_message]

while True:
    response = model(conversation)      # the model thinks
    conversation.append(response)

    if response.has_tool_calls:
        results = run(response.tool_calls)   # the harness acts
        conversation.append(results)
        # loop again -- the model sees the results and continues
    else:
        break   # no tool calls == the model is done

show(response.text)   # final answer to the user

That's it. That's the agentic loop.

Every turn, the model sees the entire conversation so far -- your request, every command it has run, and every result. It picks the next action. The harness carries it out. Repeat until the model stops asking for tools.

Let's trace our "fix the failing test" example through it:

You: "fix the failing test"
Model: calls run_bash("pytest -q")
Harness: returns 1 failed ... assert add(2, 2) == 5
Model: calls read_file("math_utils.py")
Harness: returns the file contents
Model: calls edit_file(...) to fix the bug
Harness: applies the edit
Model: calls run_bash("pytest -q") again
Harness: returns 5 passed
Model: no more tool calls -- replies "Fixed it. The bug was..."

Ten steps, one loop. No magic -- just a model reacting to real output, one turn at a time.

Why this design is so powerful

Once you see the loop, a lot of things make sense:

It reacts to reality. The model doesn't guess whether the test passed -- it sees the output and adjusts. That feedback is what makes it feel smart.
Tools are pluggable. Want the agent to query a database or hit an API? Add a tool to the menu. The loop doesn't change. (This is exactly what MCP servers do -- they extend the menu.)
Safety lives in the harness, not the model. Before step 7 actually writes to disk, the harness can pause and ask you for permission. The model can request anything; the harness decides what's allowed. That separation is your safety layer.

Skills: giving the agent a playbook without bloating its brain

So far the model knows two kinds of things: what's baked into its training, and whatever's in the current conversation. But real work needs specifics -- your team's deploy steps, a tricky migration checklist, the exact way your repo wants commits formatted.

You could paste all that into every conversation. But then it's competing for space in that limited context window, every single session, whether you need it or not.

Skills are the fix. A skill is just a packaged set of instructions for a particular kind of task, sitting on disk until it's needed. Think of it as a playbook the agent can pull off the shelf.

The trick is how they load, and it's a pattern worth knowing: progressive disclosure.

At startup, the harness shows the model only a one-line description of each skill -- a table of contents, not the book.
When your request matches one ("deploy the service"), the model calls a tool to open that skill.
Only now do the full instructions load into the conversation -- right when they're relevant, and not a moment before.

So the agent can "know" about a hundred skills while paying the context cost of only the one it's actually using. It's the same pluggable idea from earlier, applied to knowledge instead of actions: the menu stays cheap, and the details arrive on demand.

And the best part -- a skill is just files. Here's a whole one:

---
name: deploy-service
description: "Deploy the web service to staging and run smoke tests"
---

1. Run `make build`.
2. Push the image with `make push TAG=staging`.
3. Wait for the rollout: `kubectl rollout status deploy/web`.
4. Hit `/healthz` and confirm a 200 before reporting success.

Drop that markdown file in, and you've taught the agent a new procedure -- no retraining, no changes to the loop itself.

Skills vs MCP, quickly: MCP servers add new tools (new actions the agent can take). Skills add new instructions (know-how for using the tools it already has). One grows the hands; the other grows the playbook.

The one catch: context isn't infinite

Notice that every tool result gets appended to the conversation. Read a big file? It's in there. Run a chatty command? That's in there too.

But the model can only look at a limited amount of text at once -- its context window. On a long session, all those results pile up and eventually won't fit. (This is the very problem skills are designed to sidestep.)

The fix is compaction: when the conversation gets too big, the harness summarizes the older parts to keep the important bits and drop the noise. It's the tradeoff that lets a session run for hours without the model forgetting why it started.

(That's a whole post of its own -- maybe part 2.)

The napkin, one more time

Strip away everything and a coding agent is a handful of pieces:

A model that only produces text
A set of tools it can request, with a clear contract
A loop that runs those requests and feeds results back
A permission gate in the harness that decides what's allowed
Optional skills -- playbooks the agent loads only when they're relevant

That's the whole machine. The intelligence is in the model; the agency is in the loop.

Next time you watch Claude Code fix a test on its own, you'll know exactly what's happening under the hood -- one turn at a time.

If this was useful, next part will dig into context management: how an agent keeps working long after the conversation outgrows its memory. Follow along.

The Hidden Cost of a Log Line : Sync/Async Flush and everything in Between

Mukul S — Wed, 29 Jul 2026 06:37:15 +0000

log.info("user logged in") looks free. It isn't. Behind that one line is a chain of decisions — buffer or not, flush or not, block or drop, same thread or another — and each one trades latency, throughput, and durability against the others. This post walks the whole chain, from the method call down to the bytes hitting the disk platter.

If you've ever wondered why your p99 latency has a mysterious spike, why logs vanish after a crash, or what "async logging" actually buys you, this is for you.

First, the map: facade vs. implementation

Java logging is a two-layer cake, and mixing up the layers is the #1 source of confusion.
The facade is the API your code calls. The implementation is what actually writes the bytes.

your code
   │  log.info(...)
   ▼
┌───────────────────────────────┐
│  Facade: SLF4J (or Log4j2 API)│   ← the interface you compile against
└──────────────┬────────────────┘
               │ bound at runtime
   ┌───────────┼────────────┬──────────────┐
   ▼           ▼            ▼              ▼
Logback   Log4j2 Core   java.util.logging  ...
(the engine that buffers, formats, and flushes)

SLF4J — the de-facto standard facade. Your app should log against this.
Logback — the reference SLF4J implementation. Solid, widely deployed.
Log4j2 — the performance-focused implementation, famous for its lock-free async loggers.
java.util.logging (JUL) — built into the JDK, rarely chosen on purpose.

Why the split? So you can swap engines without touching a single log. call. Everything interesting in this post — the buffering, the flushing, the async magic — happens in the implementation layer.

The anatomy of a single log call

Before we talk flushing, let's see what one log.info(...) actually does. There are five stages:

1. Level check      → is INFO enabled for this logger? (cheap, often the fastest bail-out)
2. Build LogEvent   → capture message, timestamp, thread, MDC context, maybe a stack trace
3. Filter           → run any configured filters
4. Layout / encode  → turn the event into bytes ("2026-07-28 12:00:01 INFO ...")
5. Append           → write those bytes to the destination (file, console, socket)

Stage 5 — the append — is where sync vs. async and flush-vs-no-flush live. It's also, by far, the most expensive stage, because it may touch the disk.

Pro tip already: Stage 1 is why you sometimes see if (log.isDebugEnabled()). It skips stages 2–5 when the level is off. With modern parameterized logging (log.debug("x={}", x)) the framework does this check for you before building the string — so log.debug("x=" + x) is the real anti-pattern, because the string concatenation happens regardless.

Sync logging: the default, and its two hidden layers

"Synchronous" means the append happens on your application thread. Your thread doesn't return from log.info(...) until the write is done. Simple, predictable — and where all the flush nuance lives.
Here's the subtlety most people miss: between your log statement and the actual disk, there are two separate buffers.

 your app thread
      │  writes formatted bytes
      ▼
┌─────────────────────┐
│ App-level buffer    │   ← BufferedOutputStream inside the appender
│ (e.g. 4 KB)         │      flush() empties THIS
└──────────┬──────────┘
           │ flush()
           ▼
┌─────────────────────┐
│ OS page cache       │   ← the kernel's copy, still in RAM
│ (kernel memory)     │      fsync() empties THIS
└──────────┬──────────┘
           │ fsync()
           ▼
┌─────────────────────┐
│ Physical disk       │   ← now it survives a power loss
└─────────────────────┘

Two different operations empty two different buffers, and people constantly confuse them:

flush() pushes bytes from the application buffer into the OS page cache. After a flush, another process (like tail -f) can see your log line. But it is still in RAM — a kernel panic or power loss loses it.
fsync() (via FileChannel.force()) forces the OS to write the page cache to physical storage. This is what actually makes a log line survive a crash — and it's dramatically slower. Most logging frameworks give you knobs for the first buffer and, effectively, never touch the second by default. That's the durability tradeoff hiding in plain sight. ### immediateFlush: the knob that matters most Both Logback and Log4j2 file appenders expose immediateFlush: Logback:

<appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>app.log</file>
    <immediateFlush>true</immediateFlush>  <!-- default: true -->
    <encoder>
        <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
    </encoder>
</appender>

Log4j2:

<File name="FILE" fileName="app.log" immediateFlush="true">
    <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</File>

What it does:

immediateFlush=true (the default): call flush() after every log event. Your line is in the OS cache the instant the call returns. Safe if the JVM crashes (the OS still has the bytes and will write them). But it means a syscall per log line — the throughput killer under load.
immediateFlush=false: let the BufferedOutputStream fill up (typically 8 KB) and flush only when it's full. Far fewer syscalls, much higher throughput. The cost: if the JVM dies, whatever's sitting in the app buffer (up to 8 KB of your most recent, most interesting logs) is gone. That's the fundamental sync-flush tradeoff in one sentence: flush every line for safety, or buffer for speed.

Note: Log4j2 automatically forces immediateFlush=false behavior for its async loggers — because in the async world, the fix for durability is different (more on that below).

So does sync logging survive a crash?

Depends on which crash and which flush setting:

Failure	`immediateFlush=true`	`immediateFlush=false`	With `fsync`
JVM crash (exception, OOM)	✅ safe (OS has it)	⚠️ lose app buffer	✅ safe
Kill -9 the process	✅ safe (OS has it)	⚠️ lose app buffer	✅ safe
OS kernel panic	❌ lose page cache	❌ lose more	✅ safe
Power loss	❌ lose page cache	❌ lose more	✅ safe

Almost nobody does fsync per log line — it's punishingly slow (milliseconds per call). Logs are treated as "best effort durable," and that's usually the right call. Just know the guarantee you're actually getting.

Async logging: get off the hot path

Sync logging's real problem isn't correctness — it's that your request thread pays the I/O bill. If the disk hiccups, if a log rotation stalls, if the buffer flushes at the wrong moment, that latency lands directly on the user request that happened to log at that instant. This is a classic source of mysterious p99 spikes.
Async logging fixes this by handing the log event to another thread and returning immediately:

 app thread                         background thread
     │  log.info(...)                     │
     │──── enqueue event ────►  [ queue ] │
     │  returns instantly                 │──► format + write + flush
     ▼                                    ▼
 keep serving the request         does the slow I/O

Your app thread's job shrinks to "drop the event in a queue and move on." The slow work — formatting, writing, flushing — happens on a dedicated logging thread that nobody's waiting on.
There are two very different implementations of this idea, and the difference is the whole ballgame.

Flavor 1: Logback / Log4j2 `AsyncAppender` (a blocking queue)

This is the classic approach: wrap a real appender in an async one backed by a BlockingQueue (an ArrayBlockingQueue).

<!-- Logback -->
<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
    <queueSize>256</queueSize>              <!-- default: 256 -->
    <discardingThreshold>51</discardingThreshold>  <!-- default: 20% of queueSize -->
    <neverBlock>false</neverBlock>          <!-- default: false -->
    <appender-ref ref="FILE"/>
</appender>

The three knobs that decide its behavior under stress:

queueSize — how many events can wait in line. Bigger = absorbs bigger bursts, uses more memory.
discardingThreshold — when the queue gets this full, drop lower-priority events (TRACE/DEBUG/INFO) and keep WARN/ERROR. Logback defaults this to 20% of the queue remaining. Set it to 0 to never discard.
neverBlock — what to do when the queue is completely full:
- false (default): the app thread blocks until space frees up. You keep every log, but now you've reintroduced the exact latency you were trying to avoid — async silently becomes sync under load.
- true: drop the event and move on. Latency stays flat, but you lose logs during bursts. There is no free lunch here. A bounded queue under sustained overload has only three options: block the producer, drop events, or grow unbounded (OOM). Every async logger is just choosing which of these to do, and when. ### Flavor 2: Log4j2 Async Loggers (the LMAX Disruptor) Log4j2's headline feature is Async Loggers, built on the LMAX Disruptor — a lock-free ring buffer that came out of high-frequency trading. This is a genuinely different beast from a blocking queue, and it's why Log4j2 benchmarks so aggressively.

<!-- Make ALL loggers async (system property) -->
<!-- -Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -->
<!-- Or mix: specific loggers async, rest sync -->
<AsyncLogger name="com.myapp" level="info"/>
<Root level="info">
    <AppenderRef ref="FILE"/>
</Root>

Why a ring buffer instead of a queue? Two reasons:
1. Lock-free = no contention. A BlockingQueue uses locks; when many threads log at once, they fight over that lock and serialize. The Disruptor uses a pre-allocated ring buffer and atomic sequence counters (CAS) instead of locks. Producers and the consumer coordinate via cursor positions, not mutual exclusion. At high thread counts this is the difference — Log4j2's own numbers show async loggers sustaining millions of messages/sec where a blocking queue plateaus.

        ring buffer (pre-allocated slots, reused forever)
        ┌───┬───┬───┬───┬───┬───┬───┬───┐
        │ 5 │ 6 │ 7 │   │   │ 1 │ 2 │ 3 │
        └───┴───┴───┴─▲─┴───┴───┴───┴─▲─┘
                      │               │
              consumer cursor    producer cursor
              (writes to disk)   (app threads publish here)

2. Garbage-free. The slots in the ring buffer are allocated once and reused. A queue allocates a new node per event, feeding the garbage collector. The Disruptor pre-allocates, so in steady state it creates (almost) no garbage — which means no GC pauses caused by logging. Log4j2 pairs this with a "garbage-free" layout mode that reuses StringBuilders and buffers, so you can log hard with near-zero allocation.

What happens when the ring buffer fills? `AsyncQueueFullPolicy`

Same fundamental problem as before — a bounded buffer can overflow — and Log4j2 makes the policy explicit:

Default policy: the producing thread blocks (busy-spins/waits) until a slot frees up. No lost logs, but backpressure hits your app thread.
Discard policy: drop events at or below a configured level (default: drop INFO and below) when full. Keeps latency flat, loses low-priority logs.
You can also plug in a custom policy. And there's a sharp edge worth knowing: if a log call happens on the background consumer thread itself (e.g. logging from inside a layout or an exception handler), blocking would deadlock — so Log4j2 detects this and routes it synchronously instead.

The durability twist: async can lose logs a sync setup wouldn't

Here's the tradeoff people forget. With async logging, when your app thread returns from log.error("about to crash"), that event is just sitting in a queue in memory. It hasn't been formatted, let alone written or flushed.
If the JVM crashes in the next millisecond, that error log — the one explaining the crash — is gone. With synchronous immediateFlush=true logging, the same line would have been in the OS cache and survived.
This is the cruel irony of async logging: it's fastest exactly when you're logging the most, which is often right before something goes wrong.
Mitigations:

Shutdown hooks / graceful drain. Both frameworks try to flush the queue on orderly shutdown. This handles clean exits, not kill -9 or hard crashes.
Log fatal errors synchronously. A common pattern: async for INFO/DEBUG, but route ERROR/FATAL through a synchronous, immediate-flush appender so the important stuff is durable.
Accept the loss. For high-volume, low-value logs (access logs, debug traces), losing the last few hundred on a hard crash is fine. Match the guarantee to the value of the log.

Putting numbers to it (and the benchmarking trap)

Rough ordering of throughput, slowest to fastest:

sync + immediateFlush=true        ▓▓                      (syscall per line)
sync + immediateFlush=false       ▓▓▓▓▓▓                  (buffered)
async AsyncAppender (queue)       ▓▓▓▓▓▓▓▓▓▓
async Loggers (Disruptor)         ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓

But — and every honest benchmark says this — your mileage will vary wildly, because:

Peak vs. sustained throughput are different questions. Async is fantastic at absorbing bursts. But your disk's real write bandwidth is a hard ceiling. If you sustain more log volume than the disk can drain, the queue fills and async degrades to whatever its full-policy is (blocking or dropping). Async doesn't make your disk faster — it just decouples your app from disk jitter.
Layout cost is often the real bottleneck. A fancy pattern with caller location (%class, %line, %method) forces a stack-trace walk on every log line — that can cost more than the write itself. Avoid location info in hot paths.
Microbenchmarks lie. Logging in a tight loop with nothing else running doesn't reflect a real app where the GC, the CPU cache, and other threads are all contending. The practical takeaway isn't a number — it's a decision tree.

A practical decision guide

Default for most services:
Async logging (Log4j2 Async Loggers if you can, Logback AsyncAppender otherwise), with a bounded queue sized to absorb your typical burst, and low-priority events discarded under pressure.
When you need every log line (audit, compliance, financial):
Synchronous + immediateFlush=true. Accept the throughput hit; you're buying durability. Add fsync only if you truly can't lose data on a power cut — and know it'll cost you.
When latency is sacred (trading, real-time):
Log4j2 Async Loggers with garbage-free layout and a discard policy — never let logging block a request thread, and never let it trigger GC.
A solid hybrid that covers most people:

INFO/DEBUG  → async, discard-under-pressure  (high volume, low value)
WARN/ERROR  → sync, immediateFlush=true       (low volume, high value)

You get speed for the noise and durability for the signal.

The five things to actually remember

A log call has two buffers below it. flush() empties the app buffer to the OS; fsync() empties the OS to disk. They are not the same, and only the second survives a power loss.
immediateFlush is the core sync knob. true = safe but a syscall per line; false = fast but you lose the app buffer on a crash.
Async moves I/O off your request thread — killing latency jitter — but every bounded queue must eventually block, drop, or OOM under sustained overload. Know which one yours does.
The Disruptor wins by being lock-free and garbage-free, not by magic. It beats blocking queues under high thread counts and avoids GC pauses.
Async trades durability for speed. The log explaining your crash is the one most likely to be lost in the queue. Route critical logs synchronously. Logging feels like the most boring line in your codebase. It's also one of the few that quietly touches concurrency, the memory hierarchy, syscalls, GC, and durability all at once. Now you know what that one line is really doing. 🚀 ---

What's your logging setup — sync-and-safe, or async-and-fast? And has "async logging ate my crash log" ever bitten you? Tell me in the comments.

NGINX, Actually Explained: Architecture, Config, and the Mental Model That Makes It Click

Mukul S — Wed, 29 Jul 2026 06:06:37 +0000

Most people meet NGINX as a magic file they copy-paste until the 502 goes away. This post is the opposite: a tour of the model behind that file, drawn straight from the official docs at nginx.org/en/docs. Once the model clicks, the config stops being scary.

Why NGINX is fast (and why the answer matters)

Classic web servers spawn a process or thread per connection. That's fine at 200 connections and catastrophic at 200,000 — each thread eats memory and forces the kernel to context-switch constantly. This is the famous C10K problem.
NGINX was built to dodge it. Instead of "one connection = one thread," it uses a small, fixed number of worker processes, each running a single-threaded, event-driven, non-blocking loop. One worker juggles thousands of concurrent connections by never sitting idle waiting on I/O.

That one design decision explains almost everything else about NGINX. Keep it in mind as we go.

The process model: master + workers

When NGINX starts, you get one master process and several worker processes.

          ┌─────────────────┐
          │  master process │   (runs as root, reads config, binds ports)
          └───────┬─────────┘
                  │ manages
      ┌───────────┼───────────┐
      ▼           ▼           ▼
  ┌────────┐  ┌────────┐  ┌────────┐
  │worker 0│  │worker 1│  │worker 2│   (do the actual request handling)
  └────────┘  └────────┘  └────────┘

The master process does the privileged, one-time work:

Reads and validates configuration
Binds to privileged ports (like :80 / :443)
Spawns, monitors, and restarts workers
Handles signals for graceful reloads and upgrades Worker processes do everything else — accepting connections, reading requests, talking to upstreams, sending responses. They're the hot path. A sane default is one worker per CPU core:

worker_processes auto;   # matches the number of available cores
events {
    worker_connections 1024;   # max simultaneous connections *per worker*
}

Your theoretical connection ceiling is roughly worker_processes × worker_connections. With 8 cores and 1024 connections each, that's ~8,000 connections without breaking a sweat — and that's a conservative default.

The trick that makes reloads zero-downtime

Run nginx -s reload and NGINX doesn't drop a single connection. The master:

Re-reads and validates the new config
Starts new workers with the new config
Tells old workers to stop accepting new connections
Old workers finish their in-flight requests, then exit This graceful choreography is why you can ship config changes to a busy production server in the middle of the day. --- ## The event loop: how one worker serves thousands Here's the part that trips people up. A single worker is single-threaded, yet handles thousands of connections. How? Instead of blocking on I/O, each worker asks the kernel: "tell me when any of these connections has something ready." On Linux that mechanism is epoll (it's kqueue on BSD/macOS). The worker then loops:

while (true) {
    events = wait_for_ready_connections()   // epoll_wait — the only place we "block"
    for (event in events) {
        handle(event)                        // never blocks; do a slice of work, move on
    }
}

The rule inside a worker is sacred: never block. A slow disk read or a slow upstream must not freeze the whole loop, because that loop is serving thousands of other clients. When work would block, NGINX either offloads it (thread pools for disk I/O) or parks the connection and comes back when the kernel says it's ready.
Choose the connection-processing method explicitly if you want, though NGINX auto-selects the best one for your OS:

events {
    use epoll;              # Linux
    worker_connections 4096;
    multi_accept on;        # accept as many new connections as possible per event
}

This is the whole "why is NGINX so fast" answer in one sentence: it turns I/O waiting into an event notification instead of a parked thread.

How NGINX processes a request

The docs have a dedicated page on this ("How nginx processes a request"), and it's worth internalizing. Requests are routed in two steps: which server block, then which location.

Step 1: pick the `server` block

NGINX matches the request against listen directives and the Host header (via server_name):

server {
    listen 80;
    server_name example.com www.example.com;
    # ...
}
server {
    listen 80 default_server;   # fallback when no server_name matches
    server_name _;
    return 444;                 # drop connections to unknown hosts
}

server_name matching order: exact name → leading wildcard (*.example.com) → trailing wildcard (www.example.*) → regex. First match wins.

Step 2: pick the `location`

Within the chosen server, NGINX matches the URI against location blocks. The matching rules are precise and worth memorizing:

location = /exact      { }   # 1. exact match — highest priority, stops search
location ^~ /assets/   { }   # 2. prefix match that *skips* regex if it wins
location ~ \.php$      { }   # 3. case-sensitive regex (first match wins)
location ~* \.(jpg|png)$ { } # 4. case-insensitive regex
location /             { }   # 5. plain prefix — longest match wins as fallback

The priority order is: = beats ^~ beats regex (~/~*) beats plain prefixes. Getting a 404 or serving the wrong file is almost always a location precedence surprise — this table is the cure.

The config blocks you'll actually use

NGINX config is a tree of contexts. Directives are only valid in certain contexts. Here's the shape:

# main context
worker_processes auto;
events { ... }               # connection processing
http {                       # everything HTTP lives here
    include mime.types;
    sendfile on;             # zero-copy file serving via the kernel
    upstream app { ... }     # a pool of backend servers
    server { ... }           # a virtual host
}
stream { ... }               # raw TCP/UDP proxying (databases, gRPC, etc.)

Serving static files

server {
    listen 80;
    server_name static.example.com;
    root /var/www/html;
    location / {
        try_files $uri $uri/ =404;   # try file, then dir, else 404
    }
    location /assets/ {
        expires 30d;                  # aggressive caching for hashed assets
        add_header Cache-Control "public, immutable";
    }
}

try_files is the workhorse here — it's how SPAs get their "always fall back to index.html" behavior:

location / {
    try_files $uri $uri/ /index.html;   # client-side routing friendly
}

Reverse proxy to an app server

This is probably why you're here. NGINX sits in front of your Node/Python/Go app and forwards requests:

upstream backend {
    server 127.0.0.1:3000;
    server 127.0.0.1:3001;
    # load balancing: round-robin by default
    # least_conn;                # send to the worker with fewest active conns
    # ip_hash;                   # sticky sessions by client IP
    keepalive 32;                # reuse upstream connections
}
server {
    listen 80;
    server_name api.example.com;
    location / {
        proxy_pass http://backend;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
        proxy_set_header Connection "";    # required for upstream keepalive
    }
}

Those proxy_set_header lines matter more than they look. Without them, your backend sees NGINX's IP as the client, loses the original scheme, and can generate broken redirects. This four-line block is the single most copy-pasted (and most misunderstood) snippet in NGINX land.

Load balancing methods, at a glance

Method	Directive	Use when
Round robin	(default)	Backends are roughly equal
Least connections	`least_conn;`	Requests have uneven duration
IP hash	`ip_hash;`	You need sticky sessions
Weighted	`server ... weight=3;`	Backends have different capacity

HTTPS in about ten lines

The docs' "Configuring HTTPS servers" page boils down to this:

server {
    listen 443 ssl;
    server_name example.com;
    ssl_certificate     /etc/nginx/ssl/example.com.crt;
    ssl_certificate_key /etc/nginx/ssl/example.com.key;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    ssl_session_cache   shared:SSL:10m;   # reuse handshakes across connections
}
# Redirect all HTTP to HTTPS
server {
    listen 80;
    server_name example.com;
    return 301 https://$host$request_uri;
}

`ssl_session_cache` is the performance sleeper: TLS handshakes are expensive, and caching sessions across connections cuts CPU noticeably on busy sites.

Rate limiting: cheap insurance

NGINX can shield your backend from bursts and abuse before a request ever reaches your app. It uses a leaky bucket algorithm:

http {
    # define a zone: 10MB of state, 10 requests/sec per client IP
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    server {
        location /api/ {
            limit_req zone=api burst=20 nodelay;   # allow short bursts of 20
            proxy_pass http://backend;
        }
    }
}

There's a sibling directive, `limit_conn`, for capping concurrent connections per client. Both live entirely in NGINX — no app code, no extra service.

The commands worth memorizing

nginx -t                 # test config syntax before applying — do this ALWAYS
nginx -s reload          # graceful reload, zero dropped connections
nginx -s quit            # graceful shutdown (finish in-flight requests)
nginx -s stop            # fast shutdown (drop everything now)
nginx -T                 # dump the full, resolved config (includes all `include`s)
nginx -v                 # version
nginx -V                 # version + build flags + configured modules

Rule of thumb: never `reload` without running `nginx -t` first. A syntax error caught by `-t` is a non-event; the same error hitting `reload` on some setups can take the server down.

Debugging: read the logs like a local

Two logs, two jobs:

nginx
http {
    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    'rt=$request_time uct="$upstream_connect_time" '
                    'urt="$upstream_response_time"';
    access_log /var/log/nginx/access.log main;
    error_log  /var/log/nginx/error.log warn;
}

Adding $request_time and $upstream_response_time to your access log is the fastest way to answer "is it NGINX or is it the backend that's slow?" — a question you will be asked during an incident.

For deep debugging, a build with `--with-debug` unlocks `error_log ... debug;`, which traces the request lifecycle in detail. That's covered in the docs' "A debugging log" page.

The mental model, in five lines

If you remember nothing else:

Master configures, workers serve. Privilege separation + zero-downtime reloads.
One worker, one event loop, thousands of connections. Never block the loop.
Requests route in two steps: server block, then location — and precedence is not intuitive, so learn the table.
Contexts nest: main → http → server → location. Directives are context-scoped.
nginx -t before every reload. Always. Everything in the config file is a consequence of these five ideas. Once you see the event loop behind the directives, NGINX stops being a black box and starts being the most predictable thing in your stack. ---

Where to go next in the docs

Beginner's Guide — a hands-on first config
How nginx processes a request — the routing internals in depth
Admin Guide — proxying, load balancing, compression, caching
Module reference — every directive, grouped by module (ngx_http_core_module, ngx_http_proxy_module, ngx_http_upstream_module, …)

All at nginx.org/en/docs.

Found this useful? Drop your gnarliest NGINX config gotcha in the comments — the location precedence ones are always a good time. 🚀

Request	What happens
1	`System + U1 + A1 + U2` → everything written to cache
2	`... + A2 + U3` → System→U2 read from cache, A2+U3 written
3	`... + A3 + U4` → System→U3 read from cache, A3+U4 written