DEV Community: Athreya aka Maneshwar

GFS Cost More Than 50 Cents a Run. Now Each Run Costs 4 Cents

Athreya aka Maneshwar — Sat, 26 Sep 2026 18:02:42 +0000

Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

The bill was 50 cents a run.

Not a month. Not a thousand runs. One run.

Paste a design proposal in, get back a grounded review with precedent from our shelf of books and internal postmortems, and watch half a dollar evaporate.

Comments warn about API key store isolation quirks

Athreya aka Maneshwar

Sep 22

Cheap RAG in Go with Gemini File Search: no vector DB, two calls, one hosted store

#go #ai #rag #gemini

14 min read

Part 1 was the happy version of this story: a Go binary, Gemini's hosted File Search, no vector database, and a free tier that made the whole thing look like a free lunch.

Then the free tier ran out and the paid rates arrived, and the free lunch turned out to be a tasting menu.

Five people on the team, several runs a day each, and a number that scaled with how hard the tool was thinking.

This post is what we replaced it with, what we measured, and the one component we built, benchmarked, and then deleted.

The word "free" was doing a lot of work

Here is the thing the pricing page tells you about File Search, and it is all true.

Storage is free. Query-time embeddings are free. You pay once at indexing time.

Here is the thing it does not put in bold.

The search is performed by a model. It is a tool the model calls mid-answer. Which means the retrieved chunks land in that model's context as ordinary input tokens, and the model reasons its way through them before replying.

And on Gemini 3.5 Flash, reasoning costs $9.00 per million output tokens.

Look at that output row.

9,529 tokens out, of which 8,493 were the model thinking. For a call whose entire job is "find the relevant passages and hand them over."

We paid a reasoning model to reason about which paragraphs to copy.

The first fix was structural and it helped a lot. The draft step used to do its own search, which meant three drafts meant three searches. Splitting search out so one search feeds every draft took a run from about $0.70 to about $0.12.

Six times cheaper, and still the wrong shape.

Because the expensive part was never the searching. It was the thinking attached to it.

Retrieval does not need a brain.

It needs an index, a similarity function, and a tiebreaker. Every one of those is a thing you can run on a laptop while it charges.

What we swapped in

A local Chroma store, embedded with Qwen3-Embedding-0.6B, searched with dense vectors plus BM25, fused with Reciprocal Rank Fusion.

Every model call moved to DeepSeek V4 Flash on Atlas Cloud at $0.14 in and $0.28 out per million.

Roughly ten times cheaper per token than what we were on, with a 1M context window and JSON mode.

The retrieval service is Python, running as a child process of the Go binary, on a port nobody else talks to. It starts with make run and dies with it.

I did try to talk myself out of that boundary.

Go has no mature CUDA story. The honest route is exporting the model to ONNX and linking a Go runtime through cgo, matching driver and CUDA and cuDNN versions exactly. PyTorch is the path a million people have already debugged, including on WSL2, where CUDA passthrough breaks in creative ways.

Adding a process boundary was cheaper than adding a whole new class of "works on my machine."

Chunking is where the accuracy actually lives

Everyone talks about which embedding model to pick. Almost nobody talks about what you feed it, and that is where the wins were.

Our corpus is 71 markdown files, and most of them are books that used to be PDFs. PDFs converted to markdown are full of things that are not text.

The cleaner strips the converter's own footer, lines that are only a page number, picture-text blocks, and hyphenated line breaks that split a word across two lines.

The one that surprised me is running headers.

A converted book repeats the chapter title at the top of every single page, usually promoted to a markdown heading.

Left in, it shreds the text into confetti, and a passage about paper reactors comes back with "Ship Project and Civilian Power" wedged into the middle of a sentence.

The rule that fixed it is embarrassingly simple: a short line that appears five or more times in one file is page furniture, not prose.

Then the chunking. About 300 words, hard cap 450, split on sentence boundaries, with two sentences carried into the next chunk so a quote that straddles a boundary survives in at least one piece. A real section heading starts a new chunk, once the current one has enough in it to stand alone.

And then the part I would steal even if you take nothing else from this post.

Embed more than you store.

The chunk you store is the text the draft is allowed to quote. The string you embed has a header glued on top of it:

EMBED_MODEL = "Qwen/Qwen3-Embedding-0.6B"

# what goes into the embedding, per chunk
f"{title} ({kind}) › {heading_path}\n\n{text}"

# and the query side gets an instruction, because Qwen3-Embedding is
# instruction-tuned on queries only. documents are embedded as they are.
QUERY_PROMPT = (
    "Instruct: Given an abstract pattern or principle, retrieve historical cases, "
    "documented examples, and named principles from books and essays that show "
    "the same pattern\nQuery: "
)

A chunk in the middle of chapter nine might only say "he decided otherwise."

With the header, that vector still knows it is Rickover, in a book about Rickover, in a section about paper reactors. Without it, it is a pronoun floating in space.

The query instruction is the Part 1 idea, "search with the pattern, not the post," pushed one layer down into the embedding itself. The corpus is cases. The queries are patterns. Saying so out loud to a model that was trained to listen costs nothing.

Dense finds the paraphrase, BM25 finds the name

Vector search is great at "these two paragraphs mean the same thing" and oddly bad at "this paragraph contains the word Rickover."

Keyword search is the reverse.

So we run both, take 40 candidates each, and fuse them.

flowchart TD
    Q[one query from call 1] --> DN[dense top 40, Qwen3-Embedding]
    Q --> BM[BM25 top 40, title + text]
    DN --> RRF[Reciprocal Rank Fusion, k=60]
    BM --> RRF
    RRF --> SEL[keep the best 4 per query]
    SEL --> CAP{2 chunks from this file already?}
    CAP -- yes --> SKIP[skip, so no book dominates]
    CAP -- no --> KEEP[keep it]
    KEEP --> RR[round-robin merge, 3 queries]
    SKIP --> RR
    RR --> DUP{near-duplicate of one picked?}
    DUP -- yes --> DROP[drop: a post synced twice]
    DUP -- no --> OUT[12 passages go to the draft]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef dense    fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef lex      fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
    classDef good     fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef bad      fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a

    class CAP,DUP decision
    class Q start
    class DN,RRF dense
    class BM,SEL,RR lex
    class KEEP,OUT good
    class SKIP,DROP bad

def hybrid(self, query: str, k: int = CANDIDATES) -> list[tuple[int, float]]:
    score: dict[int, float] = {}
    for ranked in (self.dense(query, k), self.lexical(query, k)):
        for r, i in enumerate(ranked):
            score[i] = score.get(i, 0.0) + 1.0 / (RRF_K + r + 1)
    return sorted(score.items(), key=lambda x: -x[1])[:k]

That is the whole fusion. Six lines.

The reason it works is that it never compares the two scores. A cosine similarity of 0.82 and a BM25 score of 14.3 have nothing to say to each other. Ranks do.

A document at position 3 in both lists beats one that is first in a single list, and the constant k (60 is the number the literature settled on) keeps the top of each list from steamrolling everything else.

BM25 indexes the title and heading alongside the text, same as the embedding header does, so naming a book in your query actually finds pages from that book.

A few rules keep the final twelve honest. At most two chunks per file per query, so one 400-page book cannot fill every slot. Queries merge round robin, so each of the three contributes its best passage before any of them gets its fourth.

And anything with a word-overlap above 0.6 against something already picked gets dropped, because our blog corpus has a couple of posts that got synced twice and they were politely returning themselves as two independent sources.

The reranker that got fired

Standard advice says: retrieve broadly, then rerank with a cross-encoder. So we did that, with bge-reranker-v2-m3.

Then a run took six and a half minutes and I went looking.

The retrieval service's own log had it in one line: search: 3 queries -> 8 chunks in 117.5s.

Three queries times 40 candidates is 120 cross-encoder forward passes on a 4 GB GTX 1650 that is also drawing the desktop. It was not thrashing. It was just honest work on unfit hardware.

Before ripping it out, we measured. The golden set builds itself out of finished sessions: take Call 1's retrieval queries, pair them with the files the accepted draft actually cited, and you have a retrieval test built from real usage rather than from vibes.

The reranker won on MRR and hit@4. It genuinely put better passages nearer the top.

It also did not change recall@12 at all, and recall@12 is the only number with a consumer.

The draft call gets all twelve passages in its prompt. It reads all twelve. There is no top-4 cutoff downstream, no truncation, nothing that treats passage 1 differently from passage 9.

So the reranker was spending 108 seconds improving an ordering that nothing downstream reads. It was optimising a metric we had accidentally chosen because it appears in every retrieval paper, not because our pipeline consumed it.

Out it went, with the reasoning written into the top of retriever.py so the next person does not "fix" its absence.

And the honest caveat, which lives there too: this is one golden session. A harder query might genuinely need reranking to pull the right passage into the top twelve. The code is in git history, the eval is a make target, and when the golden set is fat enough to mean something we will run it again.

Measure before you delete. Also measure before you keep.

Two hours of embedding, four laptops

3,639 chunks at roughly 1.8 seconds each on a shared 4 GB GPU is about two hours, which is about one hour and fifty minutes more than anyone wants to wait.

But embedding is deterministic and the corpus splits cleanly by file. So it parallelises across people, not just across cores.

flowchart TD
    C[71 files, 3,639 chunks] --> S[shard_files: disjoint quarters]
    S --> M[four laptops, --shard i/4]
    M --> R{sha256 AND chunk count match?}
    R -- yes --> SK[already embedded, skip]
    R -- no --> EM[embed, halve the batch on OOM]
    EM --> G[commit the shard, hand it back]
    SK --> G
    G --> I[rag/integrate.py]
    I --> V{same embedding model everywhere?}
    V -- no --> X[refuse: mixed vectors lie quietly]
    V -- yes --> O{every file in exactly one shard?}
    O -- in two --> X
    O -- in none --> W[warn, merge what arrived]
    O -- yes --> F[copy vectors into db/chroma]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef work     fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef box      fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
    classDef good     fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef bad      fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a

    class R,V,O decision
    class C start
    class M,EM work
    class S,G,I,SK box
    class F good
    class X,W bad

# on four different machines, one quarter each
uv run rag/ingest.py --shard 1/4 --out db/chroma-shards/1
uv run rag/ingest.py --shard 2/4 --out db/chroma-shards/2
# ... then, once everyone commits their shard back
uv run rag/integrate.py db/chroma-shards/*

The merge does no embedding at all. It checks that every shard used the same embedding model, that no file landed in two shards, and that no file landed in none, then copies the vectors into one store.

Those checks are not paranoia. Mixed embedding models do not crash. They return confidently wrong neighbours forever, which is a far worse failure than a stack trace.

Building this also shook out a real bug in the incremental logic. Resume was deciding "already done" by comparing the file's sha256 against what was in the store.

A run killed halfway through a file leaves that file's sha perfectly correct and its chunk count short, so it would have been marked done forever, silently missing half a book.

The fix is one AND: sha256 and chunk count both have to match.

Then we committed the finished store. db/chroma is 69 MB in git, which is nothing, and it means nobody else on the team ever embeds anything. Clone, run, search.

Now that retrieval is free, spend it on drafts

The nice thing about killing your most expensive call is that the cheap calls get interesting.

A draft is now a few tenths of a cent. So instead of drafting once and retrying on failure, the pipeline fires three or four drafts at once at temperature 0.7 and lets them race.

Each reply gets resolved and checked in Go as it lands. Drafts that fail the checks are rejected on the spot. The first one that passes goes on to the review call. Only if all of them fail does the batch retry, with the closest draft's failures appended to the prompt.

Every draft is kept and shown, including the rejected ones with the checks they failed, because "here are four attempts and why three of them were bad" is more useful to the person reading than one draft and a shrug.

This did produce one genuinely dumb bug, which parallelism made much more likely.

In one run the first batch produced a draft that passed every check. The reviewer then asked for a revision. Eight redrafts later, none of them passing, the pipeline shipped the closest failing redraft.

It had a passing draft in hand and threw it away for a worse one. The fix is the obvious fallback: if no redraft passes, ship the draft that already did, with the reviewer's notes attached.

Retries are cheap. Losing work you already paid for is not.

The check that replaced trusting the grounding metadata

Handing the passages in ourselves unlocked the thing I actually care about.

When a hosted search tool returns grounding metadata, you know which chunks the model looked at. You do not know that the words it put in quotation marks are in any of them.

Now the pipeline can prove it.

Every history.sources[].passage in the output has to appear word for word in a chunk from the file it names. Same for an authority's exact_words when it cites one of the given files. Markdown, line wrapping and quote style are normalised away first, and a ... marks an omission, with the remaining pieces required to appear in order.

A failure is not a warning. It is a check failure, exactly like a length violation or an unresolved law citation, and it feeds the same retry loop that everything else does.

This is the difference between "the model had access to the right book" and "the model quoted the right book correctly," and only one of those is worth showing a reviewer.

What the bill looks like now

One real run, end to end, with 15 model calls including 12 drafts across three batches:

210,976 input tokens, 55,285 output tokens
$0.045, about ₹4.3
retrieval: 0.5 seconds of it, and none of the money

Against $0.70 for the same pipeline shape on hosted search. Fifteen times cheaper, and the expensive part is now the part that does the actual writing, which is how it should be.

Two honest footnotes on that number.

Atlas reports most of each draft's input as cached, and we charge every input token at the full rate, so $0.045 is a ceiling, not an estimate.

And latency went up, not down. Retrieval dropped from 117 seconds to half a second, but DeepSeek thinks hard before each draft, so a full run is still two to four minutes.

We bought cost, not speed. For "paste a design doc, come back with a coffee," that is the right trade. For a chat box it would be the wrong one.

What I would steal

If your retrieval is a model call, you are paying reasoning prices for a lookup.

Pull it onto your own machine, spend the effort on cleaning and chunking rather than on model selection, embed a contextual header you never show anyone, fuse dense and lexical on ranks instead of scores, and check the quotes rather than trusting them.

Then build the eval before you build the clever part. Ours told us to throw the clever part away, which saved 108 seconds a search and a permanent dependency we did not need.

The best component in this system is the one that is not in it.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

Cheap RAG in Go with Gemini File Search: no vector DB, two calls, one hosted store

Athreya aka Maneshwar — Tue, 22 Sep 2026 07:12:46 +0000

Every team I have been on has the same shelf.

Nine engineering books somebody swears by, a couple of long essays that get linked in every third design review, and sixty-odd internal blog posts and postmortems that nobody rereads.

About 1.4 million tokens of "we already learned this once."

I wanted a tool where you paste a design proposal, an ADR, a postmortem draft, and it comes back with: here is what is actually being proposed, here is the pattern underneath it, and here are the three places on the shelf where we, or someone smarter, already ran into this exact shape.

Grounded. With citations. Not "an LLM read your doc and had feelings about it."

The obvious build is a vector database, an embedding pipeline, a chunker, a reranker, and a weekend.

I built it with none of those, in Go, on Gemini's free tier, and the retrieval side of it costs nothing to run.

This post is about how, and about the four things that bit me on the way.

RAG without owning the R

Gemini has a thing called File Search.

You create a "store", upload documents into it, and Google chunks them, embeds them, and indexes them.

Then you attach that store to a normal generateContent call as a tool, and the model searches it by itself, mid-answer, and hands you back the chunks it used as groundingMetadata.

No pgvector. No Pinecone bill. No embedding model to pick and then regret.

The pricing is the part that made me sit up.

Storage is free. Query-time embeddings are free. You pay once at indexing time, at embedding prices, and then the chunks the model pulls in are billed as ordinary context tokens on the call you were already making.

On the free tier the store caps at 1 GB. My entire shelf, every book and every post converted to markdown, is 5.3 MB.

So the architecture is embarrassingly short.

One Go binary. SQLite via modernc.org/sqlite, so no cgo. A REST client for Gemini with no SDK, because I wanted to log the exact request and response bodies verbatim, and SDKs love to hide those.

Two model calls per question. One hosted store. That is the whole thing.

Step zero: getting books into markdown without losing the spaces

Before any of the clever parts, the corpus has to exist as text, and this cost me an evening.

The first tool I reached for was markitdown. It gave me headings, sort of, and it also gave me this:

Theubiquityoffrustrating,unhelpfulsoftwareinterfaceshasmotivateddecadesofresearch

Every word in the PDF glued to its neighbour. The embedding model does not know what "Theubiquityoffrustrating" is, and neither does the retrieval.

pdftotext fixed the spacing and threw away every heading, so a 300-page book became one undifferentiated scroll.

The one that won was pymupdf4llm. It looks at font sizes to decide what is a heading, keeps bold and italics, and its text extraction handled the spacing correctly on the same PDF.

uv run --with pymupdf4llm python -c "
import pymupdf4llm, pathlib
pathlib.Path('out.md').write_text(pymupdf4llm.to_markdown('in.pdf'))
"

Forty-four real headings out of one essay, versus zero.

Headings matter more than they look like they should here, because File Search chunks on whitespace with a token budget. A chunk that starts at a heading is a chunk that means something on its own. A chunk that starts mid-sentence in a wall of text is noise with an embedding attached.

I put the conversion behind make process-data so nobody on the team has to rediscover this. EPUBs, PDFs, and a sync of our blog repos, all into one post_processed_data/ tree of markdown.

The store belongs to the key, not to you

This is the first thing that bit me, and it is not in the big print.

A File Search store lives inside the Google Cloud project behind the API key that created it.

Create a store with key A, try to search it with key B from a different project, and it does not error in a helpful way. It just is not there.

That matters the moment you have more than one key, which on the free tier you will, because the per-minute caps are real and the fix everyone reaches for is "rotate across a few keys."

You cannot rotate a tool-attached call across keys. The store pins you.

So keys got roles.

keys/store.txt is the short list. Each key in it owns a complete copy of the corpus in its own store. Ingest uploads to every one of them. The search call tries store 0 with key 0, and only on a quota or auth failure falls through to store 1 with key 1.

keys/rotate.txt is the long list, for any call with no tool attached: the analysis call, the corpus-index summaries. Those genuinely do not care which key answers, so they rotate on 429, 401, 403, and anything 5xx.

Two things I got wrong before I got them right:

503 is not a key problem. Gemini returns 503 "high demand" when Google is busy, and switching keys does nothing except burn another key's quota. So the client waits and retries the same key a couple of times before falling over.

The key goes in the x-goog-api-key header, never the URL. Put it in the query string and the first transport error prints your key into your own logs. Ask me how I know.

Yes, two stores means the one-time indexing cost happens twice. It is the price of the search call never dying on a single quota, and at embedding prices for 5 MB, it is coffee money.

Ingest is a diff, not an upload

The ingest step is where "just upload the folder" turns into an actual program.

Every file gets a sha256. A manifest in SQLite records path -> sha -> document id for each store. Then:

flowchart TD
    A[walk data/**/*.md, sha256 each] --> B{path in manifest?}
    B -- no --> U[upload to store]
    B -- yes --> C{sha changed?}
    C -- no --> S[skip, unchanged]
    C -- yes --> D[delete old doc id] --> U
    U --> P[poll the indexing operation until done]
    P --> M[upsert manifest: path, sha, doc id]
    A --> R{manifest path missing on disk?}
    R -- yes --> X[delete doc from store, drop manifest row]
    R -- no --> S
    M --> I{set of path+sha changed?}
    S --> I
    X --> I
    I -- no --> K[corpus index unchanged]
    I -- yes --> G[summarize new files with flash-lite, one line each] --> J[store corpus_index]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef net      fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef local    fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
    classDef idx      fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a

    class B,C,R,I decision
    class A start
    class U,D,P,X net
    class S,M,K local
    class G,J idx

Unchanged files are skipped. Changed files have their old document deleted first, then get re-uploaded. Files that vanished from disk get deleted from the store.

Each upload is a multipart POST that returns a long-running operation, and you poll it until done before you trust the document id. Four uploads run in parallel. Serial, a 71-file corpus takes about ten minutes. Parallel, a few.

The chunking is set per upload, and I landed on 400 tokens with 60 of overlap. The docs' example is 200 and 20, which for a book felt like reading through a letterbox.

meta := map[string]any{
    "displayName": displayName,
    "chunkingConfig": map[string]any{
        "whiteSpaceConfig": map[string]int{
            "maxTokensPerChunk": 400,
            "maxOverlapTokens":  60,
        },
    },
}
// multipart/related: part 1 is this JSON, part 2 is the markdown bytes
// POST /upload/v1beta/{store}:uploadToFileSearchStore
// with X-Goog-Upload-Protocol: multipart

The last box in that flowchart is the corpus index, and it is small but it is the thing that makes the next section work.

For every file, a one-sentence summary from the cheapest model available (gemini-3.5-flash-lite), keyed by the file's sha so it is only ever generated once. All the lines get joined into one block, "path, summary", that gets pasted into the analysis prompt.

That way the model deciding what to search for knows what is actually on the shelf. It aims at sources that exist instead of guessing.

Search with the pattern, not the post

This is the idea in the post that I would keep if I had to throw everything else away.

Embedding search finds things that sound alike.

Paste a proposal that says "we should standardize on one MCP transport across all our agents" straight into retrieval, and you get back every chunk that contains the words "MCP", "agents", and "standardize". Which is your own recent blog posts about MCP and agents.

That is not precedent. That is a mirror.

What you actually want is the shape of the situation with the nouns removed.

"Fragmented, incompatible implementations, then one open standard, then mass adoption."

Search the shelf with that, and the browser wars come back. Rickover on standardizing the nuclear navy comes back. A 2,300-year-old Legalist essay on uniform law comes back.

Same pattern, different clothes. That is what a reviewer with thirty years of reading brings, and it is what the model cannot do if you let it search with the raw text.

So the pipeline never lets it.

Call 1 gets the lawbook, the corpus index, and the proposal, with no tools attached. It returns JSON: the motive, what is actually happening, who the actors are, the generalization, and two or three retrieval queries derived from that generalization.

Call 2 gets the analysis from call 1, the File Search tool pointed at the store, and the instruction to search with those queries and then draft.

The raw document is in call 2's context, so the draft can quote it. But the search terms came from call 1, and they are about the pattern, not the subject.

{
  "systemInstruction": { "parts": [{ "text": "...the lawbook..." }] },
  "contents": [{ "role": "user", "parts": [{ "text": "PROPOSAL:\n...\n\nANALYSIS FROM CALL 1:\n{...}\n\nSearch the corpus with the retrieval queries above, then reply with the JSON contract." }] }],
  "tools": [{ "fileSearch": { "fileSearchStoreNames": ["fileSearchStores/abc123"] } }],
  "generationConfig": { "responseMimeType": "application/json", "temperature": 0.7 }
}

The response carries candidates[0].groundingMetadata.groundingChunks, each with the file's display name and the retrieved text. Those get shown to the user, in full, next to the draft. If the "authority" the review leans on is not in that list, it did not come from the shelf, and the checks catch that.

Every instruction is a law, every reply cites its laws

There is no free-text system prompt anywhere in this thing.

The prompts are an AgentLaws lawbook: a folder of markdown where every instruction the model sees is a numbered law, grouped into chapters, versioned in git, and compiled once at startup.

The model is required to return applied_laws, the numbers it relied on, and the pipeline resolves each one back to file:line in the lawbook.

A number that does not resolve fails the run. Which sounds pedantic until the first time the model confidently cites law 7.2.3 in a lawbook with six chapters.

The practical win is that "why did it do that" has an answer that is a file and a line number, and "make it stop doing that" is a pull request against a markdown file, not archaeology in a Go string.

The output contract is the last thing in the prompt, because that is where the model weights it most. Laws about tone and structure go first, the JSON schema goes last.

Trust, but run the checks

The model also fills a checks block in its own reply, a self-audit. I do not trust it, but it is useful as a second signal.

Before that gets read, a small pile of deterministic checks in plain Go runs on every draft: length bounds, no markdown, at most one question, no "see point N" references, every cited law resolved.

flowchart TD
    A[call 2: search + draft, JSON mode] --> B{reply parses as JSON?}
    B -- no --> F[resend once without JSON mode, extract first object] --> C
    B -- yes --> C[resolve every cited law number to file:line]
    C --> D{all citations resolve?}
    D -- no --> R
    D -- yes --> E[deterministic checks: length, no markdown, one question max, distinct openers]
    E --> G{checks pass?}
    G -- no --> R{attempts left?}
    R -- yes --> H[append the failures to the user message] --> A
    R -- no --> S[ship the draft with failures listed]
    G -- yes --> V[call 3: reviewer pass, verdict + objections]
    V --> W{verdict?}
    W -- ship --> O[return JSON with sources and applied laws]
    W -- revise, rounds left --> H
    W -- revise, no rounds --> S

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef llm      fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef local    fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
    classDef bad      fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef good     fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a

    class B,D,G,R,W decision
    class A,F,V llm
    class C,E,H local
    class S bad
    class O good

A failure gets appended to the user message as a plain list, "the previous attempt failed these checks, fix them", and call 2 runs again. Two retries, then the draft ships anyway with the failures listed, because a draft with a warning is more useful than a spinner that gave up.

One box in there deserves its own sentence: JSON mode plus a tool is not guaranteed. responseMimeType: application/json together with File Search worked most of the time and then, occasionally, returned prose with a JSON object somewhere in it. The fix is boring: if the reply does not parse, resend once without JSON mode and pull the first {...} out of the text. Both attempts get logged.

The bug that made me split the database

The tool is used by five people, and each of them runs it against their own SQLite file, so history is per person.

The first version put everything in that one file: sessions, steps, and the ingest manifest.

Spot it?

A new teammate makes a new database. New database, empty manifest. Empty manifest means every file on disk is "new", so all 71 get uploaded again. And because the manifest is empty, ingest does not know the old document ids, so nothing gets deleted.

The store now has two of everything. Do it five times and search quality quietly rots, because every query returns the same chunk five times and crowds out the second-best hit.

The fix is one sentence: the manifest is a property of the store, not of whoever is asking questions today.

So there are two databases now. corpus.db holds the store names, the manifest, and the corpus index, and every session database reads from it. alice.db holds Alice's sessions and nothing else.

Delete corpus.db and you get a full re-ingest. Delete alice.db and nothing about the corpus changes.

What the free tier actually gives you

Since "cheap" is in the title, the honest numbers, as of when I built this.

Models. Flash only. gemini-3.5-flash for the real calls, gemini-3.5-flash-lite for the throwaway summaries. Pro returns a 429 with limit: 0 on free keys. A paid key unlocks it with one env var, and I have not needed to.

Rate limits. Per-minute caps are the thing you hit, not per-day. Ingest backs off with a growing wait when every key is rate limited, and the UI shows each key fallover as it happens instead of hiding it behind a spinner.

Storage. 1 GB per store. My 5.3 MB corpus does not register.

Cost per question. Two Flash calls, sometimes three with the review pass, around 10k tokens in and 2k out. The retrieved chunks are already counted in that. On the free tier, zero. On a paid key, well under a cent.

Latency. 30 to 90 seconds end to end, most of it call 2 doing the search and the draft. That is fine for "paste a design doc, get a review" and would be wrong for a chat box.

The catch. On the free tier, Google may train on what you upload. My shelf is published books and public blog posts, so I do not care. If your corpus is your customer contracts, read the terms before you make ingest.

What I would tell you to steal

If you have a shelf, and a question shaped like "have we seen this before", you do not need a vector database to answer it.

You need markdown with real headings, a hosted store you diff against instead of re-uploading, keys with roles because the store pins you to one, and a first model call whose only job is to turn the question into the pattern underneath it.

The rest is checks and logging, and the checks are the part you will thank yourself for.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

How to Count 100 Billion Things in 12 Kilobytes

Athreya aka Maneshwar — Wed, 16 Sep 2026 12:23:43 +0000

An interviewer asks you a question that sounds almost insultingly easy.

How many unique visitors did the site have today?

You say: easy, I'll hash every request ID into a set. HashSet<String>, one entry per unique visitor, done before the coffee's cold.

Then they say: it's a hundred billion requests.

Your hash set just quietly asked the OS for about a terabyte of RAM, and the OS just as quietly said no.

You could shard it. Spin up forty machines, split the ID space, merge partial sets at query time.

That works. It is also an entire distributed system you now have to run, just to answer "how many different people showed up."

There's a much smaller way to answer this, and it doesn't even need to remember the visitors.

The naive fix, and the bill for it

A hash set is exact because it remembers everything.

Every single unique ID gets a slot, forever, because that's the only way to know for certain you haven't seen it before.

That's also exactly why it doesn't scale. Memory grows linearly with the number of uniques, no matter how you slice it, shard it, or compress the keys.

If your product only ever has a few hundred thousand daily uniques, this is a complete non-problem. Use the hash set, go home.

But once you're talking billions, "remember everything" stops being an engineering decision and starts being a bet against your cloud bill.

The way out isn't a bigger hash set. It's giving up on "exact" entirely, on purpose, in exchange for something that fits in your CPU's L2 cache.

The trick: rare things are evidence of scale

Here's the actual insight, stripped of the mechanism around it.

Hash every ID into a long, effectively random bit string.

Now look at how many zeros it has at the start, before the first 1 bit shows up.

Since each bit is a coin flip, the odds are simple: 50% of hashes start with at least one zero, 25% start with at least two, 12.5% with at least three, and so on. Each extra leading zero you demand halves how many hashes will have it.

Flip that around and it becomes a counting trick.

If you've hashed a handful of IDs and none of them start with three zeros, that's unremarkable, you'd expect that from 8 items. But if you've seen a hash that starts with six zeros, that's a 1-in-64 event. Seeing it once is weak evidence you've hashed something on the order of 64 items, because you'd need to try roughly that many random hashes before one that rare shows up.

So: hash every incoming ID, and keep a running max of the longest leading-zero streak you have ever seen. That single number, "longest streak seen so far," gives you a rough estimate of 2^streak unique items.

This idea goes back to Flajolet and Martin's original 1985 probabilistic counting paper, and the modern form is called HyperLogLog, from Flajolet, Fusy, Gandouet and Meunier's 2007 paper.

Here's the single-counter version, in about ten lines, so you can see exactly how weak it is on its own:

import random

def leading_zeros(x, bits=32):
    if x == 0:
        return bits
    return bits - x.bit_length()

def single_counter_estimate(n_items):
    max_streak = 0
    for _ in range(n_items):
        h = random.getrandbits(32)
        max_streak = max(max_streak, leading_zeros(h))
    return 2 ** max_streak

# run it a few times on the same n and watch the estimate swing wildly
for _ in range(5):
    print(single_counter_estimate(10_000))

Run that a handful of times against the same n_items and the estimates will swing by 2x, 4x, sometimes more in either direction. One lucky hash with an extra-long streak, or one unlucky run without one, and the whole estimate moves.

A single coin flip streak is just too noisy to trust. HyperLogLog doesn't use one counter, it uses thousands, and averages them in a very specific way.

Averaging away the noise

Instead of one counter, take the first few bits of each hash and use them to pick one of m buckets, say m = 16384. The remaining bits of the hash get the leading-zero treatment from before, and the result updates that bucket's own running max.

You now have thousands of tiny, independent "longest streak I've seen" estimators, each looking at a different slice of the ID space. Average them, and the noise from any one unlucky or lucky streak gets smoothed out by the other 16,383 buckets.

There's one more wrinkle worth knowing. HyperLogLog uses the harmonic mean across buckets, not the arithmetic mean. A plain average gets wrecked by a single bucket that got a freakishly long streak, since 2^streak grows exponentially, one outlier bucket can dominate a normal average completely. The harmonic mean punishes large outliers far more than small ones, which is exactly the failure mode you need to guard against here.

flowchart TD
    A[ID arrives] --> B[Hash it to a uniform bitstring]
    B --> C[First p bits pick a bucket]
    C --> D[Count leading zeros in the rest]
    D --> E{Longer streak than<br/>this bucket has seen?}
    E -->|Yes| F[Store it as the bucket's max]
    E -->|No| G[Discard, bucket keeps its max]
    F --> H[m buckets, each holding one max streak]
    G --> H
    H --> I[Harmonic mean across all buckets]
    I --> J[Bias-correct for small and huge counts]
    J --> K[Cardinality estimate, ~2% error]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef step fill:#6ea8ff,stroke:#2f5fbf,color:#1a1a1a
    classDef store fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef result fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a

    class A start
    class E decision
    class B,C,D step
    class F,G,H store
    class I,J,K result

With m buckets, the standard error works out to roughly 1.04 / sqrt(m). Plug in 16,384 buckets and you land at just over 0.8% expected error, which is where Redis's implementation sits.

That error bound doesn't depend on whether you counted a million items or a hundred billion. It only depends on m, the number of buckets, which is a number you pick up front.

That's the whole trade. You fix your error tolerance once, at design time, by choosing m, and the memory cost stays flat forever after. A few small corrections handle the edges: linear counting kicks in when very few buckets have been touched yet, so a tiny actual count doesn't get wildly overestimated, and a large-range correction avoids hash collisions skewing things once you approach 2^32 distinct items.

Somebody already built this into your database

The best part of HyperLogLog isn't the estimate, it's that two of them merge for free.

Since each bucket is just "the max streak seen," merging two HyperLogLogs means taking the elementwise max of their buckets. No replay, no recomputation, no access to the original IDs at all. That's why it works so well for things like "daily uniques" that you also want to roll up into "weekly uniques."

Redis has had this built in for years, as three commands:

# add IDs as they arrive, one PFADD per event
PFADD visitors:2026-09-16 user_881 user_204 user_991

# get the estimated cardinality, ~12KB per key no matter how big it gets
PFCOUNT visitors:2026-09-16

# merge daily counters into a weekly one, no replaying the day's events
PFMERGE visitors:week-38 visitors:2026-09-12 visitors:2026-09-13 visitors:2026-09-14 visitors:2026-09-15 visitors:2026-09-16

Redis's PFADD docs put the standard error at 0.81%, using Google's HyperLogLog++ paper for the small-and-large range corrections on top of the original algorithm.

It's not just Redis either. BigQuery's APPROX_COUNT_DISTINCT, Presto and Trino's approx_distinct, and Spark's approx_count_distinct are all the same idea wearing a SQL function's clothes. Every "unique visitors" widget on every analytics dashboard you've ever glanced at is very possibly running this exact algorithm under the hood, right now, while you read this sentence.

When you actually still want the hash set

None of this means throw away exact counting. It means know which question you're answering.

If the number needs to be exact, billing a customer per API call, deduplicating a payment, deciding whether a user already redeemed a coupon, HyperLogLog is the wrong tool. It cannot tell you whether one specific ID has been seen, only roughly how many distinct ones have.

If your cardinality is small anyway, a few thousand, a few hundred thousand, a plain hash set fits in memory with room to spare and gives you an exact answer for free. Reach for HyperLogLog when the count itself is the product, "how many uniques," "how many distinct IPs," "how many distinct search terms," and the scale is big enough that remembering every item stops being realistic.

That's the whole pitch. You trade a guarantee you never actually needed for a memory bill you can actually afford, with error bounds you get to choose in advance.

Next time somebody says "a hundred billion" in an interview, you now have a much better answer than "shard it."

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

MVC, MVP, MVVM, MVVM-C, VIPER: one pattern wearing five outfits

Athreya aka Maneshwar — Tue, 15 Sep 2026 17:47:49 +0000

Every mobile team eventually has the same argument.

Someone says the ViewController is 3,000 lines long and needs to be broken up.

Someone else says "just use MVVM," like that phrase alone fixes anything.

A third person mentions VIPER and the room goes quiet, because everyone has a VIPER war story.

I went down this rabbit hole recently and realized something that should have been obvious years ago: MVC, MVP, MVVM, MVVM-C and VIPER are not five different ideas.

They're one idea, wearing five different amounts of clothing.

The one shape underneath all of them

Strip away the acronyms and every pattern is arguing about the same three things.

A View, which is the face of the app. It renders pixels and captures your taps.

A Model, the brain, which owns the business logic and the data.

And a translator in between, whose entire job is making sure the View and the Model never talk to each other directly.

That translator is the part that keeps getting renamed. Controller. Presenter. View-Model. Everything else is a debate about how much power to give it, and who else gets to share the job.

Model-View-Controller is genuinely old, coming out of Smalltalk work at Xerox PARC in the late 1970s, which makes it nearly as old as the personal computer itself.

It was built to separate "what the data is" from "what's on screen," and for a single-window desktop app in 1978, that was already a big improvement over one giant blob of code that did everything.

To see where each pattern actually differs, let's run the exact same tiny feature through all five: a user tapping their profile picture to pick a new one.

MVC: everything reports to one controller

MVC connects the View and the Model through a Controller that does all the coordinating.

You tap the picture, the View tells the Controller, the Controller updates the Model, and once that's done the Controller turns around and tells the View to refresh.

It reads clean on a whiteboard, and honestly it's a fine choice for a small app. That's not a knock, it's the actual selling point.

class ProfileController {
  onPhotoPicked(image) {
    model.setAvatar(image)      // update the model
    view.render(model.avatar)   // tell the view to refresh
  }
}

The catch is that this Controller doesn't stay this small. Every new feature wants to talk to the Model somehow, and the Controller is the only door in the building.

Six months later it's not a controller, it's a lobby that every department in the company has to walk through.

MVP: the Presenter takes the homework off the View

MVP answers the bloated-controller problem by handing UI logic to a dedicated Presenter, and telling the View to do nothing but draw.

Tap the photo, the View notifies the Presenter, the Presenter updates the Model, formats whatever comes back into something display-ready, and pushes that formatted result straight into the View.

The View in MVP is deliberately dumb, which is a compliment here. A dumb View is a View you can unit test the Presenter against without spinning up any UI at all, since the Presenter only ever talks to a thin interface the View implements.

That testability is the entire reason teams reach for MVP. Not because it's fancier than MVC, but because "does the right thing happen" becomes a question you can answer without a simulator or a device.

MVVM: the View stops asking, and starts just knowing

MVVM swaps the Presenter's manual "update the view" call for two-way data binding between the View and a View-Model.

Pick a new photo, the View pushes that change into the View-Model through binding, the View-Model persists it to the Model, and when the Model's data changes, that change flows back to the bound View property automatically. No explicit refresh call, anywhere.

This is the pattern that plays nicest with reactive frameworks, because reactive frameworks are basically data binding with better marketing.

class ProfileViewModel {
  @Observable var avatar: Image   // View binds to this directly

  func onPhotoPicked(image: Image) {
    avatar = image        // View updates instantly via binding
    model.persist(image)  // no explicit "refresh the view" call anywhere
  }
}

Less boilerplate, definitely. The tradeoff shows up later, when a bug means tracing a chain of automatic notifications instead of reading a linear function call, which is a genuinely different debugging skill.

MVVM-C: somebody still has to decide where the app goes next

MVVM never says who's in charge of moving between screens, and by default that job quietly lands on the View-Model, which is exactly the thing MVVM was trying to keep simple.

MVVM-C fixes that by adding a Coordinator that sits above the binding triangle and owns navigation on its own. In our example, that's the move from the profile screen to the image picker and back, including whatever save logic happens in between.

Same View, View-Model, Model relationship as plain MVVM. The only new thing is that "where do we go next" has its own dedicated home, instead of leaking into whichever View-Model happened to trigger the transition.

VIPER: five jobs, five files, no exceptions

VIPER takes the "give everyone exactly one job" idea about as far as it can go.

View displays and forwards user actions. Interactor owns the business logic. Presenter prepares data for the View and talks to the Interactor. Entity is the raw data, the Model equivalent. Router owns navigation.

Tap the photo: View tells Presenter, Presenter tells Interactor to do the actual work, Interactor manages the Entity, Presenter formats the result back for the View, and if a screen change is needed, Router handles it.

protocol PhotoInteractorProtocol {
  func updateAvatar(_ image: Image)
}
protocol PhotoPresenterProtocol {
  func didPickPhoto(_ image: Image)
  func didUpdateAvatar(_ image: Image)
}
protocol PhotoRouterProtocol {
  func closePhotoPicker()
}

Notice there's no shared base class holding this together, just protocols. That's the whole VIPER pitch: maximum separation, maximum testability, and yes, maximum file count for updating a single profile picture.

It earns its keep on genuinely large apps with multiple teams working the same codebase. On a five-screen app it's mostly ceremony.

So which one do you actually pick

None of them are wrong, they're just answers to different problems.

MVC for a small app where shipping fast matters more than architecture purity.

MVP once you need the mediator to be testable in isolation but don't need data binding.

MVVM once your framework is reactive and binding removes real boilerplate.

MVVM-C once that same app also has navigation flows worth centralizing.

VIPER once the app and the team are both big enough that "everyone has one job" stops being overhead and starts being the thing that keeps ten engineers from stepping on each other.

flowchart TD
    A[Starting a new client app] --> B{Small app, small team?}
    B -->|Yes| MVC[MVC: simplicity wins, ship it]
    B -->|No| C{Need the mediator unit-testable?}
    C -->|Yes, no data binding needed| MVP[MVP: Presenter tests clean in isolation]
    C -->|Need reactive data binding| D{Complex navigation flows too?}
    D -->|No| MVVM[MVVM: binding kills boilerplate]
    D -->|Yes| E{Large app, many teams?}
    E -->|Not yet| MVVMC[MVVM-C: Coordinator owns navigation]
    E -->|Yes| VIPER[VIPER: five single-job pieces, max modularity]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef pick     fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef heavy    fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a

    class A start
    class B,C,D,E decision
    class MVC,MVP,MVVM pick
    class MVVMC,VIPER heavy

The pattern isn't the architecture. It's just how much ceremony your team is willing to pay for, in exchange for how much chaos it's trying to avoid.

Pick the smallest one that still lets you sleep at night, and upgrade only when the pain is real, not hypothetical.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

The ffmpeg Pipeline Nobody Explains

Athreya aka Maneshwar — Mon, 14 Sep 2026 11:34:19 +0000

ffmpeg is the one CLI tool everyone uses and most doesn't understand

Every developer has run this exact incantation at some point in their life, copy-pasted from a Stack Overflow answer from 2014, with zero idea what it does:

ffmpeg -i input.mov -vcodec h264 -acodec mp2 output.mp4

It works. You move on. You never ask what -vcodec actually does, or why some conversions finish in a blink and others chew your CPU for ten minutes.

Turns out there's a genuinely elegant pipeline hiding under the hood, and once you see it, half of ffmpeg's flag soup starts making sense on its own.

ffmpeg was created by Fabrice Bellard back in the year 2000, and the name is a mashup of "fast forward" and MPEG, the Moving Picture Experts Group, the folks behind most of the video formats you've heard of.

It's not just a CLI toy either.

It's the encode/decode engine quietly running inside Chrome, Blender, YouTube, Vimeo, and roughly half of every video-adjacent tool you've ever used.
Source: ffmpeg.org.

The pipeline nobody tells you about

Here's the thing that clicked for me: ffmpeg isn't one operation, it's a pipeline, and every flag you pass just tweaks one stage of it.

Walk through it left to right:

Input — your file lands, in.mp4, whatever.
Demux — a demultiplexer splits the container into its separate streams. Your "video file" was never one thing, it's a video track, an audio track, maybe subtitles, all interleaved into one file for convenience.
Decode — each stream's compressed packets get decoded into raw, uncompressed frames. This is the CPU-hungry part.
Filter — optional. This is the only stage that actually touches pixels or audio samples: brightness, contrast, scaling, adding subtitles, drawing a waveform. Skip it and nothing changes.
Encode — raw frames get compressed back down into packets, in whatever codec you asked for.
Mux — a multiplexer interleaves the encoded streams back into one output container.
Output — out.mkv, done.

ffmpeg -i in.mp4 out.mp4 runs the whole thing with sane defaults. ffprobe just runs the first couple of stages and prints what it finds, no encode required, which is why it's instant.

With over 100 codecs supported, that same seven-stop pipeline is how ffmpeg decodes, encodes, transcodes, filters, and plays basically any multimedia file that exists. Source: ffmpeg.org/about.html.

The flag that changes everything: `-c copy`

Once you see the pipeline, one thing jumps out immediately: decode and encode are the only expensive stages. Demux and mux are just bookkeeping, shuffling bytes around.

So what happens if you skip decode and encode entirely?

ffmpeg -i input.mp4 -c copy output.mkv

-c copy tells ffmpeg "don't touch the streams, just repackage them." It demuxes, then immediately muxes into the new container.

No decode, no encode, no quality loss, because the actual video bytes never change, they just get moved into a different box.

This is why converting .mp4 to .mkv is instant, but converting .mp4 to .webm is not: mp4 and mkv can both hold H.264 video, so it's a pure repackage.

webm wants VP8/VP9/AV1, a codec mp4 usually isn't carrying, so ffmpeg has no choice but to actually decode and re-encode every frame.

Once you've internalized that, most of ffmpeg's common recipes stop being magic incantations and start being obvious:

ffmpeg -i input.mov -c copy -ss 00:00:30 -t 00:00:10 clip.mov

Trim ten seconds starting at 0:30. Since we're not changing codecs, -c copy keeps it instant. Same file, smaller slice.

Need to glue several clips together? List them in a text file and hand it to the concat demuxer:

ffmpeg -f concat -safe 0 -i list.txt -c copy joined.mp4

Still -c copy. Still no re-encode. It's the same pipeline principle, just applied to multiple inputs instead of one.

When you actually need the expensive path

Sometimes copy isn't an option, because you're deliberately changing the pixels or the codec, not just the wrapper. That's when -vf (video filter) and real encode flags come in:

ffmpeg -i input.mp4 -vf "scale=1280:720" -r 30 -b:v 2M -c:v libx264 output.mp4

-vf scale resizes, -r sets the frame rate, -b:v sets the video bitrate, -c:v libx264 picks the encoder. Every one of these forces a real decode → filter → encode pass, which is exactly why this version is slow and -c copy isn't.

Subtitles go through the same filter stage. Got an .srt file? Convert it to .ass first, then burn it in with -vf subtitles, since -vf is the only stage in the whole pipeline that's allowed to touch a frame.

Here's the decision tree I actually keep in my head now, roughly:

flowchart TD
    A[I have a media file and ffmpeg] --> B{Just changing container, same codecs?}
    B -->|yes| B1[-c copy]
    B -->|no| C{Need a smaller file or a different codec?}
    C -->|yes| C1[-c:v libx264 -c:a aac]
    C -->|no| D{Just trimming a section?}
    D -->|yes| D1[-ss start -t dur -c copy]
    D -->|no| E{Joining multiple clips together?}
    E -->|yes| E1[concat demuxer + -c copy]
    E -->|no| F{Changing resolution, framerate or bitrate?}
    F -->|yes| F1[-vf scale, -r, -b:v]
    F -->|no| G{Burning in subtitles?}
    G -->|yes| G1[srt to ass, then -vf subtitles]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start    fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef action    fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a

    class A start
    class B,C,D,E,F,G decision
    class B1,C1,D1,E1,F1,G1 action

Every branch of that tree is the same seven-stage pipeline, just with a different subset of stages actually doing work.

The rest of the toolbox

Two more binaries ship alongside ffmpeg and are worth knowing exist:

ffprobe — inspects a file and dumps its metadata: codecs, resolution, duration, bitrate, stream count. It's the tool for "wait, what actually is this file" before you commit to a slow encode.
ffplay — a minimal media player built on the same libraries, for when you just want to preview something from the terminal without opening a full video app.

And under all three sits a stack of libraries, libavcodec, libavformat, libavfilter, and friends, that other software links against directly rather than shelling out to the CLI. That's the actual reason ffmpeg ended up powering Chrome's media playback and Blender's video editor: it was never really "a command line tool," it was a media engine that happened to ship a command line tool as its front door.

Why this is worth knowing

None of this is trivia for its own sake.
Once the pipeline is in your head, ffmpeg's entire flag surface stops being a wall of options to memorize.

You start asking "which stage am I actually touching" before you type a command, and the answer tells you whether it'll take a second or a coffee break, and whether you even need to.

That's the whole trick.
The tool has a hundred flags, but only one shape underneath all of them.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

Seven Patterns That Decide If Your AI App Survives 10,000 Users

Athreya aka Maneshwar — Sat, 12 Sep 2026 20:30:43 +0000

Here's a fun fact that nobody puts on the slide at the AI meetup.

Your agent can be perfect. The prompt can be tuned within an inch of its life. The eval score can be a smug 94%.

None of that matters the moment your product gets popular.

One user hits your FastAPI endpoint, the agent runs, the model answers, everybody's happy.

Ten thousand users hit it in the same five minutes, and suddenly the model is the least interesting part of your outage.

The bottleneck moved. It always moves. It moved from "can the model answer this" to "can the system around the model survive being asked."

That system is what this post is about. Seven patterns, one request, and the boring infrastructure work that decides whether your AI product works for a demo or for a Tuesday afternoon in production.

Zoom out one level

Picture the AI service you already built: FastAPI route, an agent workflow, a vector store, tracing, evals, the whole thing.

That entire box becomes one small rectangle in a bigger picture starting today.

Nothing inside that rectangle changes. FastAPI still handles the route, the agent still calls the model, tracing still tells you what happened.

What's new is everything around it, deciding who gets in, what waits, what fails safely, and how much capacity exists in the first place.

Pattern 1: the API gateway is your reception desk

An API gateway sits in front of your application and handles traffic before it ever reaches your route handlers.

It checks who's calling, rejects garbage requests before they cost you compute, tags each one with a request ID, and routes it to the right internal service.

Think of it as reception for a big office building. One door in, one person checking where you need to go, and you never have to know which floor anything is actually on.

One distinction that trips people up: this is not the same as a model gateway. An API gateway manages traffic coming into your app from users. A model gateway manages calls going out to OpenAI, Anthropic, or whoever's serving your model. Same word, opposite direction.

Pattern 2: rate limiting decides how much work gets to start

A gateway controlling who can knock on the door still lets in more valid traffic than you can serve. The next question isn't "who's allowed in," it's "how much work do we let start."

Plain rate limiting counts requests per user per minute. Fine for a CRUD API. Not fine for an LLM app, where request count barely correlates with cost.

One call classifies "yes" or "no" in ten tokens. Another retrieves eight documents, calls three tools, and streams back four paragraphs. Same "one request" on your dashboard, wildly different bill.

So AI systems widen the definition to admission control: cap requests, cap input tokens, cap output tokens, cap concurrent model calls, whatever actually predicts cost in your system.

When you're over budget, the honest answer is an HTTP 429 with a Retry-After header, the standard response for telling a client to back off instead of quietly queuing infinite work. That's called back pressure: push the slowdown back to whoever's asking, before it becomes your problem.

Pattern 3: cache what's actually safe to reuse

A gateway and a rate limiter, and you've still got a pile of accepted requests all asking suspiciously similar questions.

If five hundred people ask about the same public doc today, the first request should do the retrieval and embedding work, and the other 499 should get it basically for free.

Redis in front of embeddings, retrieval results, or common model responses buys you real latency and cost wins.

The catch, and it's a big one for anything touching an LLM: caching an answer is a claim that it's still correct and still safe for whoever's asking.

A private answer generated for one user is not a cache entry, it's a leak, if it ever gets served to somebody else. A cached fact from three months ago about pricing or an API limit is not "reused work," it's a wrong answer with good latency.

Every cache entry needs an expiry, a scope (user, tenant, permission level), and an invalidation path tied to the source changing. Skip any of those three and the cache stops saving you money and starts costing you incidents.

Pattern 4: let the work that can wait, wait

Caching removes duplicate work. It doesn't remove the different work, and there's a lot of AI-adjacent work that genuinely doesn't need to finish before you respond to the user: document ingestion, embedding generation on upload, a long report, an email.

That's what a durable queue is for. Take the order, hand the customer a ticket number, let the kitchen cook at the kitchen's pace instead of the counter's.

A few details that separate "we added a queue" from "we added a queue that actually holds":

Bound the backlog. An unbounded queue doesn't prevent an outage, it just delays it and makes it bigger when it finally lands.

Route failures somewhere visible. A job that keeps failing should land in a dead letter queue for a human to look at, not retry silently forever and quietly eat your worker capacity.

Make retried actions idempotent. At-least-once delivery means a worker can see the same job twice. If "process this job" means "charge this card" or "send this email," processing it twice is a bug wearing a queue's clothing.

Separate lanes for separate work. One giant document-ingestion job parked in the same queue as your live chat's background tasks will happily starve the thing your paying customers are waiting on.

def process_job(job_id, payload):
    if redis.setnx(f"processed:{job_id}", 1):
        redis.expire(f"processed:{job_id}", 86400)
        do_the_actual_work(payload)
    # already processed once, safely a no-op the second time

Pattern 5: stop one broken dependency from taking the site with it

Now the harder failure mode. Not "the dependency is down," which fails fast and cleanly, but "the dependency is slow," which is much worse.

A dependency that's fully down refuses your call in milliseconds. Your worker's free again instantly, checkout fails cleanly, the rest of the site keeps serving.

A dependency that's slow holds that same worker for thirty seconds while it decides whether to answer. Do that across your whole pool and a random product page dies for a problem it never even called.

Three tools handle this, in order:

Timeouts. Decide up front how long you're willing to wait. No timeout means every slow dependency gets to set your latency for you.

Retries, with backoff and jitter. A brief retry after a random-ish delay handles transient blips. AWS has the definitive writeup on why the jitter matters: without it, every client backs off on the same clock and retries in a synchronized wave that looks exactly like the outage you were trying to survive.

The circuit breaker, for when it's not transient. Martin Fowler's original writeup frames it exactly like the breaker in your fuse box: too many failures and the circuit trips open, calls stop reaching the broken dependency, and everyone's spared the wait.

Closed is normal operation. Open is refusing everything instantly while the failing dependency recovers off the hook. Half-open is the cautious part: let a handful of test calls through, and only reopen the floodgates if they succeed.

When the circuit's open, don't fill the gap with a guess. Serve a tested fallback model, a verified cached answer, or an honest "this feature's briefly unavailable." Graceful degradation means reduced functionality, never a confident wrong answer.

And keep failing dependencies from sharing a resource pool with healthy ones — separate connection pools per dependency, a pattern usually called a bulkhead, named for the same reason ships have them: one flooded compartment shouldn't sink the whole hull.

Pattern 6: spread the load across more than one copy

Every one of those live requests is still landing on a single instance of your core AI service, and one healthy server still has a ceiling.

Run several copies, put a load balancer in front, and it hands each request to whichever healthy instance has room.

Worth separating from the gateway, since people conflate them: the gateway decides which service a request should reach. The load balancer decides which copy of that service handles it. Some managed products bundle both jobs into one offering, but they're still two different decisions.

The thing that makes running several copies safe is keeping them stateless. Durable conversation history, workflow checkpoints, job status — put those in Postgres. Short-lived shared state goes in something like Redis. Then any instance can pick up any request, and a retried call resumes from a saved checkpoint instead of restarting the whole agent loop from scratch.

Pattern 7: autoscaling adds the capacity the rest bought you room to add

Everything above buys time. Eventually, if traffic keeps growing, distributing the load across existing copies stops being enough, and you actually need more of them.

Autoscaling adds or removes instances as demand shifts, so you're not paying for a night's worth of GPU capacity at 3am for zero users.

The part worth getting right is what signal drives it. CPU alone is a bad proxy for a GPU-backed model server, since you can be maxed out on the accelerator while CPU idles. Better signals: queue depth, how long the oldest task's been waiting, active requests, or GPU utilization directly.

New capacity also isn't instant. A stateless API pod might come up in seconds. A model server loading weights into GPU memory can take minutes. If your traffic pattern is predictable (the daily 9am spike, say), keep some capacity warm ahead of it rather than reacting cold.

This is also why autoscaling comes last in the list, not first. More servers don't fix a cache serving stale answers, a retry storm with no jitter, or a queue with no bound. Control the work first. Add capacity second.

Walking one request through the whole thing

A user's request hits the API gateway, which gives it one controlled entrance and routes it toward the right service.

The rate limiter checks whether this user, and this kind of request, is within budget.

The load balancer picks a healthy instance of the core AI service.

That instance checks the cache first: if the answer is safe to reuse, it comes back immediately.

If the request needs an answer now, it stays on the live path. If it's kicking off longer background work, it drops into the queue for a worker to pick up.

Timeouts and circuit breakers guard every call the instance makes outward, to the model provider, the vector store, any tool. The autoscaler is watching the whole system and adjusting instance count as it goes.

And inside that one small rectangle, nothing about the actual AI work changed: FastAPI handles the route, the agent runs its loop, the model generates the answer, tracing records what happened, evals score whether it was any good.

That last point is the one worth sitting with. Metrics tell you if the system's fast. Traces tell you where a request spent its time. Evals tell you if the answer was actually worth sending.

None of these seven patterns touch that last question, and they were never meant to. Scaling a wrong answer faster is not a win, it's just a faster wrong answer reaching more people.

If you ever get a system design question about this in an interview, resist the urge to start naming seven cloud products. Start with the request: how much traffic, what needs to answer immediately versus what can wait, and which dependency is going to be the first one to have a bad day. Add a pattern only when it's solving a problem you can point to.

The model was never the bug. The system that decides who gets to ask it something, and what happens while it's thinking, always was.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

How Uber Knows Your Driver Is 7 Minutes Away

Athreya aka Maneshwar — Fri, 11 Sep 2026 20:24:22 +0000

Your Uber says the driver is 7 minutes away.

They show up in 7 minutes.

That is not a lucky guess, that is one of the more quietly insane systems in consumer tech, and it is worth taking apart.

Step 1: chop the world into segments

Uber does not think about roads, it thinks about road segments.

A single road gets cut into a handful of pieces, and globally Uber is tracking around 100 million of these segments.

Each segment has a number attached to it: how long it takes to cross, right now.

To get from your driver to you, the routing engine finds the fastest path through these segments and adds up the crossing times along the way.

That sum is your ETA.

Simple enough, except that number, "how long it takes to cross this segment", is not a constant.

It is 20 seconds at 2am and 2 minutes at 6pm on a Friday, on the exact same 200 meters of road.

Step 2: don't ask Google, ask your drivers

The obvious move is to buy this traffic data from someone who already maps the whole planet.

Uber doesn't. Uber measures it.

Every driver on the platform is already pinging their location every 4 seconds, because that's what the app needs to do anyway.

That stream is basically free traffic telemetry. Every time a driver crosses a segment, Uber now knows, to the second, how long that segment just took.

Multiply that by every active driver and you get a live traffic sensor network that nobody had to install a single camera for.

Step 3: the part that actually needs machine learning

Here's the catch. Live measurements only tell you what a segment did in the past few minutes.

Your ETA needs to know what it's going to do while your driver is still en route to you, which could be 15 minutes from now.

So in 2022, Uber shipped a deep learning system called DeepETA, and it does not just average recent history.

If a normally fast road has a fresh accident on it, DeepETA trusts the live signal over the historical pattern.
If a quiet backstreet has no recent driver on it at all, DeepETA infers its state from the busier roads around it instead of guessing blind.

It refreshes these forecasts for every segment, for the next 3 hours, every few minutes, and answers roughly 2 million forecast requests a second, making it one of the busiest models running inside Uber. (Uber Engineering: DeepETA)

flowchart LR
    A[Driver GPS pings<br/>every 4s] --> B[Live segment<br/>crossing times]
    B --> C[DeepETA<br/>forecasts 3h ahead]
    C --> D[Routing engine sums<br/>segments on your path]
    D --> E[Correction model<br/>trained on real trips]
    E --> F[The ETA on<br/>your screen]

    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef chip   fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef accel  fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a

    class A start
    class B,D chip
    class C,E accel
    class F start

And there's one more pass after all that. The routing engine's segment-summed number goes through a second model, trained on millions of completed real trips, whose only job is to catch and correct the places where the physics-based sum tends to be systematically wrong (a stop sign nobody accounted for, a left turn that always takes longer than it looks).

The payoff

Shipping DeepETA improved long-trip arrival accuracy by 6%.

Uber estimates that alone is worth around $100 million a year in gross bookings, because an ETA people trust is an ETA people don't cancel on.

Next time your ETA ticks down without drama, that's 100 million road segments, a live sensor network made of other people's cars, and a model answering 2 million questions a second so a number on your screen can be boring.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

Down Is Kind, Slow Is Fatal: Circuit Breakers and the Three Links Around Them

Athreya aka Maneshwar — Thu, 10 Sep 2026 20:15:05 +0000

Picture an e-commerce store.

Checkout calls the payment service on every order. One call, one arrow on the architecture diagram, the least interesting line in the codebase.

Now break it in two different ways and watch which one hurts.

Take the payment service completely down. Every call to it is refused in about three milliseconds. Checkout returns a clean failure for that order, and every other page on the site keeps serving.

Now leave it running and answering every call correctly, only slower. 40 milliseconds becomes 30 seconds.

Checkout has nothing to react to, because from its point of view nothing failed.

On paper the second case is the healthier one. Every call is answered, correctly, with the right data.

It is also the one that takes the whole site down.

The reason is not subtle once you see it. Your service holds a worker for the entire length of every call it makes.

When the dependency is down, that worker comes back in milliseconds and moves straight on to the next request.

A slow dependency keeps the same worker for 30 seconds, and you only have so many of them.

So this whole article is about a pattern that protects you, not the service you are calling.

It works by stopping your own service from waiting.

The bill for a slow call, itemised

Every call your service makes takes a small pile of things with it while it runs.

A thread. A database connection. Some memory. A port.

It holds all of that until the call finishes, one way or the other. A slow call just holds it for longer.

You have a fixed number of each, and nothing new can start once they are all in use.

That is why the product page died in the opening scene, even though the product page never calls payment. The blocked requests were sitting on threads and connections that the rest of your system shares.

One slow dependency can saturate every resource on every one of your services in seconds.

And this is not a rare situation, because dependencies multiply. Say your service calls 30 of them, and each one is up 99.99% of the time. You need all 30.

Multiply those together and you land at about 99.7%.

That is 3 million failed requests in every billion, and over two hours of downtime a month, with all 30 dependencies behaving exactly as advertised.

The fix is to fail fast. Turn down a call you already expect to fail, instead of waiting for it to time out.

But nothing frees a slot until a call ends. So something has to decide when a call has taken too long.

Link one: the timeout

That decision is the timeout, and it is the one rule here with no exception.

Set a timeout on every remote call. Honestly, on any call that leaves your process, even one going to something on the same machine.

And there are two of them, not one. The connection timeout covers getting the socket open. The request timeout covers waiting for the answer.

Picking the number is not a vibe. AWS's Builders' Library has the method: decide what rate of false timeouts you can live with, meaning calls you cut off that would have succeeded, then set the timeout at the matching latency percentile of the service you are calling.

Their worked example accepts one call in a thousand being cut off wrongly, so they use the P99.9 latency.

Set it too high and it barely counts as a timeout at all. You are still holding the thread, the connection and the memory for the entire time you wait for a number in a config file to tell you that you are protected.

The percentile method has two places it breaks down.

It struggles with clients that carry real network latency, like calls coming in over the internet.

And it struggles with services where P99.9 sits close to P50, because then the timeout lands almost on top of the normal case.

Both of those need padding on top of the measured number.

There is one case a timeout alone does not cover, and it is the sneaky one.

A call that is slow and still succeeds never counts as a failure. It never moves a failure counter anywhere, so nothing downstream ever learns that the dependency has gone bad.

That is why a breaker needs a separate slow call rule. Resilience4j ships one: a slowCallDurationThreshold and a slowCallRateThreshold, where a call only counts toward opening the breaker once it crosses that duration.

Link two: retries, the powerful medicine

A timeout turns a hang into an error. And the first thing anyone does with an error is try it again.

That second attempt costs the server another slice of its time, and it buys that time for exactly one client.

AWS has a blunt word for this: retries are selfish.

That trade is a good one when the failure was a one-off, because the server had time to spare anyway.

It is a terrible one when the failure came from overload in the first place. You are adding load to something that is already buried, which makes it worse and holds it down long after the original problem is gone.

AWS calls retries a powerful medicine, which is a good way to hold it in your head. The right dose helps. The same substance in a larger dose does the real damage.

Backing off between attempts is the first fix, and on its own it does not go far enough.

All the calls that failed, failed at the same moment, because the dependency went bad at one instant. Back every one of them off by the same amount and they come back together too, overloading the thing a second time.

What breaks the pattern is jitter, a random delay on top of the backoff, so the attempts spread out instead of arriving in a block.

Notice what has happened to the original cause here.

Whatever knocked the dependency over is gone, and the system is still down, held there by its own traffic.

Researchers call this a metastable failure. A trigger pushes the system into a bad state, the bad state feeds itself, and removing the trigger changes nothing.

Retries make you more vulnerable to it, because they turn a small outage into an internal storm.

The part worth remembering is what this looks like from inside your own monitoring.

Clients give up waiting before the answer arrives, then send the request again.

Meanwhile your system is still finishing old calls that nobody is listening for any more.

Your throughput graph looks excellent, because work genuinely is being completed.

The number that matters is goodput, the work that reaches a caller who is still waiting, and that has gone to zero.

Link three: the breaker itself

Backoff and jitter slow the storm down, and they are worth having.

But nothing we have put on that wire so far actually stops the call.

That is the breaker's job, and the pattern is not new. It comes from Michael Nygard's Release It!, popularised precisely to prevent the cascade we have spent three sections building.

Everyone can draw the three states. Far fewer can say what number theirs trips at, or whether it can trip at all.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failure rate crosses the threshold<br/>over a real number of calls
    Open --> HalfOpen: wait duration has elapsed<br/>AND a call actually arrives
    HalfOpen --> Closed: the permitted probe calls succeed
    HalfOpen --> Open: ONE probe fails, timer restarts
    Closed: Closed<br/>calls pass through, failures counted
    Open: Open<br/>call not attempted, fails in microseconds
    HalfOpen: Half-Open<br/>a few real calls allowed through as probes

Closed is normal, and where the breaker spends almost all of its life. Calls pass straight through, failures are counted as they happen.

Open means it has tripped. The call is not attempted at all, it fails on the spot, and your application gets an exception back in microseconds instead of waiting 30 seconds. That is the fail fast idea from earlier, turned into something you can point at.

Half-open is the trial run, and it is how the breaker finds its way back. Once the reset timeout expires, it lets a limited number of real requests through to see whether the problem is fixed.

People mix this up with the retry pattern constantly, and the two are doing opposite jobs.

A retry assumes the call will succeed eventually, so it keeps trying.

A breaker assumes it will not, and stops you making a call that is likely to fail.

Three states is the teaching model. Resilience4j actually exposes six, adding METRICS_ONLY, DISABLED and FORCED_OPEN, which a person puts it into rather than a failure rate.

What actually pulls the lever

Ask most engineers what trips a breaker and they will say a count. Something like five failures in a row.

The libraries you are most likely to have installed do not work that way at all.

They measure a failure rate, and when that rate reaches your threshold, the breaker opens.

That rate is measured over a sliding window, and the window comes in two shapes. A count-based window covers the last N calls, whatever length of time those took. A time-based window covers the calls from the last N seconds, however many arrived.

A rate over a handful of calls is meaningless, because two failures out of three is 67% and also nothing.

So the rate is gated. It can only be calculated once a minimum number of calls has been recorded.

Hystrix called that gate the request volume threshold: the minimum number of requests that have to arrive in the rolling window before the circuit is allowed to trip at all.

Set it to 20. Nineteen requests come in over 10 seconds and every one of them fails. The circuit does not trip, because 19 is short of the 20 it was told to wait for.

Counting breakers do still exist. gobreaker trips by default once consecutive failures go above five. And a counter in the closed state does not pile up forever either. Azure's guidance is that the closed-state failure count is time based and resets at intervals, so the odd failure scattered across a quiet afternoon never adds up into a trip.

So four separate settings decide whether your breaker ever opens: the threshold, the shape of the window, the size of the window, and the minimum call volume.

Every one of those is a value somebody chose for you.

The defaults do not agree with each other

Start with the one most tutorials still reach for.

Hystrix is no longer in active development. It is in maintenance mode, which in Netflix's own words means they will not review issues, merge pull requests or release new versions. The last release was 1.5.18.

Netflix's own recommendation for new work is to use an active project instead, and they name Resilience4j. Internally they moved toward adaptive implementations that react to real-time performance rather than to thresholds you pick in advance.

Here are four sets of stock defaults, read the same way each time. What trips it, how long it stays open, how many calls it lets back through.

The disagreement is material rather than cosmetic.

The trip threshold runs from 10% in Polly, up to 50% in Hystrix and Resilience4j, over to six consecutive failures in gobreaker.

Time in the open state runs from 5 seconds at one end to 60 at the other.

And the half-open probe count runs from 1 to 10, which is a tenfold difference in how hard each library leans on a service that is still recovering.

Two of those defaults quietly switch the breaker off, and this is the part nobody checks.

Resilience4j needs 100 recorded calls before it evaluates a rate at all, measured over a window of the last 100 calls. Put that on a low traffic internal endpoint and it may never accumulate enough calls to evaluate anything. That breaker simply never opens.

Resilience4j also does not move from open to half-open on a timer. The transition happens when a call arrives after the wait duration has elapsed, not at the moment it elapses. On an endpoint that has gone quiet, the breaker sits open until something finally asks.

Reading your own config out loud is a genuinely useful five minutes:

resilience4j:
  circuitbreaker:
    instances:
      payment:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 100          # the last 100 calls
        minimumNumberOfCalls: 100       # <- can your endpoint even reach this?
        failureRateThreshold: 50        # percent, not a count
        slowCallDurationThreshold: 2s   # slow successes count as failures
        slowCallRateThreshold: 100      # percent
        waitDurationInOpenState: 60s
        permittedNumberOfCallsInHalfOpenState: 10   # <- the recovery dial

Half-open is where systems break a second time

Half-open exists for one reason. A service that has just come back is not at full strength, and if you hand it everything at once it falls over again.

So the breaker does not go straight from open back to closed. It opens a gate only a few calls fit through, and you choose how many.

Resilience4j describes those calls as permitted calls, and their job is to find out whether the backend is still unavailable or has become available again.

Watch what happens if one of them comes back red.

If any probe fails, the breaker assumes the fault is still there, drops straight back to open, and restarts its timer.

A single bad probe undoes the whole cooldown and puts you back at the start.

There is a way to get this wrong in the other direction too. Set the cooldown too short and the breaker keeps trying before anything has changed, so it swings between open and half-open over and over. That flapping shows up in your application as worse response times.

Put those together and you can see why half-open is where systems break twice. The first incident knocked the dependency down. The second one is your breaker's doing, because a dependency that is only halfway back got handed full traffic before it could carry it.

Link four: what do you actually return?

Every call the breaker turns away has to return something to whoever asked.

That is the fourth link, and this one is your job rather than the breaker's. The breaker only decides whether to make the call. What comes back when it says no is application logic you write.

Martin Fowler's two examples are the ones worth keeping in your head. A credit card authorization does not have to happen right now, so it can go onto a queue and be dealt with later. And missing data can often be covered by showing something slightly out of date.

Azure lists four realistic options while the breaker is open:

Degrade the feature, so the page still loads without that section.
Call something else, an alternative operation that does not depend on the broken thing.
Return a default value that means something to your application.
Report the exception to the user, which is what you get if you write no fallback at all, and is a real option rather than a failure to decide.

Think about what actually changed when you added the breaker. Your caller was going to get an error either way. The breaker changed which error it is, and how long the caller waited to receive it.

Amazon's position runs the other way, and it is worth hearing.

They avoid fallbacks for two reasons. The effectiveness of a fallback is hard to prove and hard to test. And a fallback is a mode a system only enters at the most chaotic possible moment, when things are already breaking, and switching modes right then increases the chaos.

The mechanism behind that objection is staleness in the code path itself.

Think about a fallback you wrote two years ago that almost never triggers. If there is a bug in it, or a side effect that makes the whole problem worse, nobody has looked since, and the people who wrote it have long forgotten how it worked.

Their preference turns into something you can act on this week: favour code paths that run in production continuously over ones that run rarely. If a fallback really is essential, exercise it in production as often as you can, so it behaves as predictably as the primary path.

The uncomfortable part: AWS is not sold on breakers

If fallback code is that risky, it is fair to ask whether the breaker in front of it is worth having at all.

AWS puts the objection in writing, in the Builders' Library, word for word:

Circuit breakers, where calls to a downstream service are stopped entirely when an error threshold is exceeded, are widely promoted to solve this problem. Unfortunately, circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant additional time to recovery.

There are two separate complaints packed in there.

The first is modal behavior, and it is everything we just spent this whole article building, described as a cost instead of a feature. Closed, open and half-open are three modes. Two of them are states your system is almost never in, which makes them exactly the states hardest to test.

The second is added recovery time, which is the cooldown seen from the other side. The dependency can be perfectly healthy again while your breaker is still refusing to call it, because its timer has not finished running.

What AWS ships instead, inside its own SDK, is a retry token bucket. Retries cost tokens, the bucket refills over time, and while there are tokens in it everything retries freely. Once the tokens run out, retries do not stop, they continue at a fixed rate. That behaviour went into the AWS SDK back in 2016.

The structural difference is that the bucket is continuous where the breaker is modal.

No tripped state, no cooldown, no trial run, so no rarely exercised mode for you to test. Retry capacity gets thinner as the bucket empties instead of switching off.

The bucket also never stops the original call. It limits the retries only, so the caller keeps giving the dependency a chance on every request rather than blocking a whole dependency at once.

flowchart TD
    A[Your call to a dependency<br/>is failing] --> B{What are you<br/>actually protecting<br/>against?}
    B -->|My own retries are<br/>amplifying the outage| C[Retry token bucket]
    B -->|My workers pile up on a<br/>dependency that stopped<br/>being useful| D[Circuit breaker]
    C --> E[Continuous, no modes,<br/>easy to test<br/>never blocks the first call]
    D --> F{Can it actually trip<br/>on your traffic volume?}
    F -->|No, too few calls<br/>for minimumNumberOfCalls| G[You have a decoration,<br/>not a breaker]
    F -->|Yes| H[You now own a mode<br/>you must test and alarm on]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef warn fill:#ff9a5c,stroke:#c25b23,color:#1a1a1a

    class B,F decision
    class A start
    class C,D,E good
    class G,H warn

So the rule comes down to what you are actually protecting against.

If the worry is amplification from your own retries, the bucket handles that continuously and is easier to test.

If the worry is your workers piling up against a dependency that has stopped being useful, the breaker is the thing that stops the waiting.

Choose it knowing you have added a mode you now have to test and alarm on.

Failures neither of them should ever act on

Your own bugs. A 400 is the clearest case. That is a broken request, and it will fail identically on every host you send it to. Count those and you can trip a breaker against a dependency that is completely healthy. A good breaker sorts failures by type, and can require more timeouts before tripping than outright "service unavailable" responses, because those two mean different things.

Writes with side effects. A write is only safe to retry if it is idempotent, meaning the side effect happens once no matter how many times the same request arrives. Read-only APIs usually get that for free. Resource creation APIs often do not, and that is what makes retries and fallbacks dangerous on a write path with no idempotency key.

Scoping, which is the one people get wrong quietly. One breaker per resource type stops working the moment that resource has independent providers behind it. Take a sharded data store. One shard can be completely fine while another has a temporary problem, and if you merge their errors into one breaker, it blocks calls to healthy shards while still letting calls through to the failing one.

Azure also lists five cases where a breaker only adds overhead: local in-memory resources with no network to protect, anything an ordinary retry already handles, cases where waiting out a reset introduces a delay you cannot accept, event-driven systems where failed work already lands in a dead letter queue, and systems where the platform underneath already handles recovery.

That last one deserves its own section.

Something may already be doing this for you

Everything so far has lived inside your own process, and that is not the only place this happens.

A service mesh can run a breaker for you, as a sidecar or as a capability of the platform underneath.

What it does is not quite the same thing though. Your library breaker stops calling a dependency. Envoy ejects a host, which means it pulls one bad instance out of the load balancing pool and lets the other four keep serving.

It also arrives with its own defaults, and they look nothing like the library numbers.

Envoy ejects after five consecutive 5xx responses, sweeping and re-evaluating every 10 seconds. It ejects a host for a base of 30 seconds, and it will never eject more than 10% of the fleet.

That 30 seconds is not fixed, because ejection backs itself off. The duration is the base time multiplied by how many times that host has been ejected in a row, so 30 seconds becomes 60, then 90.

Coming back is not only a matter of waiting either. A successful active health check un-ejects the host and clears its outlier counters, and clearing those resets the backoff too.

The setting that matters most mid-incident is that maximum ejection percentage. Once enough of the fleet is already out, the mesh stops ejecting, which is why you can watch a host that is clearly failing stay in the pool.

So before you tune anything in your own process, find out which of these layers is already acting on your traffic.

Possibly more than one. And all of them can be doing their job without telling you.

The part everyone skips: alarm on the state change

Fowler's requirement is straightforward. Any change in breaker state gets logged, and the breaker reveals its state so something can monitor it.

The state change is the thing you alarm on, not the errors underneath it.

Skip that wire and an open breaker becomes a silent partial outage.

Think about what your dashboards see. Every call the breaker turns away comes back fast and looks like a clean response. Your latency graph improves. Your error rate stays flat.

A good fallback does exactly the same thing to your monitoring.

The device needs a handle a person can reach, too. Operations staff should be able to trip or reset a breaker by hand, so you can force one closed when you know the dependency is back, or force it open when you know it is down.

And the person on call needs to tell those two apart. Polly makes that visible by throwing a different exception when a breaker was deliberately isolated, so a human decision never looks like the system tripping on its own.

The whole chain, one question per link

Timeout. How long are you willing to wait before you call it a failure?

Retry. How many times do you try again, and with how much jitter so your retries do not all arrive together?

Breaker, which is really three questions, because that is what a breaker is. What share of a real number of calls has to fail before you stop trying? How long do you stay stopped? How many probes do you let through on the way back?

Fallback. What do you actually return while you are stopped?

Two of those numbers carry more weight than the rest, and both are usually sitting at whatever the library shipped.

The minimum call volume decides whether your failure rate means anything at all.

The half-open probe count decides whether your recovery survives contact with real traffic.

The most useful thing you can do after reading this is open your own config and read those two lines. Then check that something alarms when the breaker changes state.

Set your timeout from a percentile rather than a round number. Jitter your retries so they do not all arrive together. Trip on a rate measured over a real number of calls. And decide what you return before the breaker ever opens.

Those four choices are the whole chain, and every number on it is yours rather than the library's.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

How Google Stores a Planet: The GFS, Explained

Athreya aka Maneshwar — Wed, 09 Sep 2026 17:32:43 +0000

Roughly 720,000 hours of video get uploaded to YouTube every day.

Call it 1,000 terabytes. Tomorrow, another 1,000. The day after, another.

To hold that you need hundreds of thousands of machines with millions of drives spinning inside them.

And here is the uncomfortable arithmetic: in a fleet that size, a drive is dying right now. Another one will die before you finish this post.

Yet not one second of anybody's cat video goes missing.

So how?

The obvious answer is money.

Google is worth a few trillion dollars, so surely they just buy the good computers, the ones that do not break.

Wrong. That computer does not exist.

Physics does not offer an enterprise tier.

Any machine you run will eventually fail, and once you have hundreds of thousands of them, failure stops being an event and becomes a background hum.

That is almost exactly how Google opened the 2003 Google File System paper: component failures are the norm, not the exception.

What YouTube runs on today is a descendant of that design.

The open source clone, HDFS, became the storage layer the entire big data industry stood on for a decade.

Let's build it up from scratch, one broken assumption at a time.

First, the file system you already have

Before we scale to a planet, look at your laptop.

Your operating system ships with a file system whose whole job is organizing bytes on a disk.

It carves your storage into equal sized blocks. Usually 4 KB each.

A 1 TB drive is therefore something like 268 million blocks, numbered from 0 all the way up.

Now save cat.png, a 12 KB masterpiece.

The file system chops it into three 4 KB chunks and drops each chunk into whichever block happens to be free. Not neatly in a row. Wherever there is space.

To ever see your cat again, it records where the pieces went in an index. Every file is a row: here are its chunks, here are the block numbers.

Click the file, the index is read, the chunks are gathered, the cat appears.

Hold that picture, because the rest of this post is the same idea with the blocks replaced by entire computers.

The file that does not fit

Now try to store YouTube.

The biggest enterprise drive you can buy today tops out around a few hundred terabytes and costs about as much as a car you would be nervous to park outside.

One day of uploads already exceeds it.

The naive fix is to build one gigantic machine and jam drives into it until the ingest fits.

Two problems, and neither is subtle.

It is one power cable away from oblivion. One outage, one fire, one clumsy technician, and every video ever uploaded is gone at once.

It does not scale. There is a hard ceiling on how many drives you can hang off a single box, and a much lower ceiling on how many requests it can serve.

So take the local file system idea and stretch it. Instead of many blocks on one machine, use many machines.

Separate boxes, separate storage, separate buildings, separate power, talking over a network.

A file comes in, gets chopped into chunks, each chunk lands on one of those machines.

Call them chunk servers, because they store chunks and later serve them. Each one holds many chunks from many different files.

Then one master plays the role the index table played. It knows every file, its chunk list, and the IP of the chunk server holding each chunk.

A client asks for a cat video. The master hands back a list of chunks and addresses. The client fetches them itself and reassembles the video.

Notice what the master is not doing: it never touches the video bytes. It hands out a map, then gets out of the way.

That single decision is why one master can serve thousands of machines without melting.

The part where everything dies

Now kill a chunk server.

A piece of the cat video just became unreachable, and a video missing a chunk is not a video. It is a buffering spinner with commitment issues.

Your instinct says this is rare. And for one machine your instinct is right. A single server might fail once a year.

But run the numbers across a fleet.

A year is about 31 million seconds. Spread one failure per server per year across a million servers and you get a failure roughly every 30 seconds, forever.

If your system needs every machine online to work, then your system is broken every 30 seconds.

This is the real GFS insight, and it is more philosophical than technical.

You do not build a reliable system by buying reliable parts. You build it by assuming the parts are garbage and designing around their funerals.

Replication buys you time

The first move is the obvious one. Keep more than one copy.

Every chunk is written to multiple chunk servers. Each copy is a replica, and the number of copies is the replication factor, which GFS sets to 3 by default.

The master now tracks, per chunk, the desired replication factor and every server holding a copy.

One server goes dark, the client just asks a different one. No drama.

But be honest about what this actually bought you.

Nothing was solved. Time was purchased.

Run long enough and all three replicas of some chunk will eventually be dead at the same time, and that chunk is gone for good. Three copies of a decaying thing is still a decaying thing.

You need the system to notice the loss and react faster than the losses accumulate.

Heartbeats, and a system that heals itself

Every chunk server sends the master a heartbeat, say every few seconds. It means nothing more than "still here."

Miss a few in a row and the master declares that server dead. It strikes it from the replica list of every chunk it was holding.

Which means those chunks now have two replicas instead of three. Under target.

So the master picks a fresh chunk server that does not already hold that chunk and tells it to copy the chunk from a server that still has a good replica.

Three again.

That loop, running constantly, is the whole trick.

Machines die at a steady rate and re-replication runs at a faster rate, so the system sits in equilibrium while the hardware underneath it quietly rots.

flowchart TD
    A[Chunk server heartbeats to master] --> B{Heartbeat received?}
    B -->|Yes| C[Mark server alive, refresh chunk map]
    C --> A
    B -->|No, 3 misses in a row| D[Declare server dead]
    D --> E[Remove it from replica list of every chunk it held]
    E --> F{Replicas below replication factor?}
    F -->|No| A
    F -->|Yes| G[Pick a chunk server without this chunk]
    G --> H[Copy chunk from a healthy replica]
    H --> I[Replica count restored to 3]
    I --> A

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef chip fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c25c1f,color:#1a1a1a

    class B,F decision
    class A start
    class C,G,H,I chip
    class D,E bad

The nice property here is that nobody pages a human at 3am for a dead disk. The dead disk is a routine input to a loop, not an incident.

But who watches the master?

You have probably spotted the hole.

Every replica of every chunk is tracked by exactly one master, and that master holds the state of the world in its memory.

Congratulations, you built a fleet of disposable machines and then hung its entire availability off one very important box.

The fix rhymes with what you already did.

The master streams every change it makes to a backup master, which sits there receiving updates and doing absolutely nothing else. This is the failover master.

Both masters heartbeat to a health check service.

Clients never hardcode a master IP. They resolve a DNS name, say master.internal.

When the health check service stops hearing from the primary master, it flips that DNS record to point at the failover, which already holds a near current copy of the state and simply takes over.

The pattern repeats at every layer: detect death with heartbeats, keep a warm copy, redirect traffic. Same song, different instrument.

Reading is easy. Writing is where it gets fun.

So far everything has been about pulling data out. Downloads are pleasant because nothing changes underneath you.

Writes are where distributed systems earn their reputation.

Here is the scenario the paper cares about, dressed in something familiar.

You share a spreadsheet with your neighbours for booking apartment parking spots. Each row is a slot. Reserving means appending your name.

Say the whole thing is one chunk, replicated across three chunk servers.

You want the spot. You ask the master for the chunk server addresses, get all three, and send your update to each of them. They confirm. All three copies are identical. Your car is parked. Beautiful.

Now your neighbour wants the same slot at the same moment.

You both get the same three addresses. You both fire off your updates.

Chunk server A receives yours first, then your neighbour's. It appends you, then them.

Chunk server B receives your neighbour's first. It appends them, then you.

The three replicas of a chunk that are supposed to be byte identical now disagree about reality, and there is no way to tell which one is right.

The paper has a word for this. Inconsistent. Data that should be the same on every server is not.

Elect somebody to be right

The root cause is that each chunk server orders updates from its own point of view. It applies what it sees in the order it sees it, cheerfully unaware that its peers saw something else.

Local time is not global truth.

The fix is to stop asking three machines to independently agree and instead appoint one of them to decide. GFS calls that server the primary for the chunk.

The master chooses it, guarantees there is exactly one primary per chunk at any moment, and remembers who it is.

Now the write flow changes shape:

You and your neighbour both ask the master for the chunk servers, and the master also tells you which one is the primary.
You both push your data to all three replicas, which buffer it without applying anything yet.
You both send a write request to the primary.
The primary picks a serial order for every mutation it received and applies them in that order locally.
It forwards that same order to the other replicas, which apply it exactly as told, discarding their own opinion about who came first.

Every replica ends up byte identical, no matter how many clients wrote at once.

Data flows to whoever is closest on the network. Order flows from a single point of authority. Separating those two is the elegant bit.

sequenceDiagram
    participant You
    participant Neighbour
    participant M as Master
    participant P as Primary replica
    participant S as Secondary replicas
    You->>M: Where is this chunk?
    M-->>You: 3 addresses, plus who is primary
    Neighbour->>M: Where is this chunk?
    M-->>Neighbour: same 3 addresses, same primary
    You->>S: push data (buffered, not applied)
    Neighbour->>S: push data (buffered, not applied)
    You->>P: write request
    Neighbour->>P: write request
    P->>P: assign serial order: You, then Neighbour
    P->>S: apply in this exact order
    S-->>P: applied
    P-->>You: done
    P-->>Neighbour: done

Note who does not appear in the hot path there. The master hands out a map at the start and then vanishes. All the heavy lifting is client to chunk server.

What GFS deliberately gave up

Every design decision above optimises the same thing: moving enormous amounts of data to enormous numbers of clients.

It is a bandwidth machine, not a latency machine.

Reading a 1 GB chunk stream is glorious. Reading one 200 byte record with a tight deadline is not what this was built for, and the designers knew it.

That is the actual lesson hiding inside GFS, and it generalises far beyond storage.

You pick the two or three properties your system genuinely needs, you optimise those relentlessly, and you take the others off the table on purpose.

A system that refuses to choose is a system that is mediocre at everything.

A question to sit with

Here is the one the paper leaves you with, and it is worth chewing on before you look it up.

Imagine one file in your GFS cluster goes viral.

A single post, a single video, and suddenly a large slice of all traffic is hammering the handful of chunk servers holding its chunks.

Those machines are drowning while the rest of the fleet is idle. The paper calls this a hotspot.

How would you spread that load?

The chunk size section of the paper has the answer Google reached for, and it is a smaller change than you would expect.

Go read it. It is fifteen pages, it is written in plain English, and it is one of the few papers that reads like somebody explaining a thing they actually built rather than a thing they wanted funded.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

7 Ways to Make Your API Faster

Athreya aka Maneshwar — Tue, 08 Sep 2026 16:42:29 +0000

Your API is slow.

You know it is slow because somebody in the product channel posted a screenshot of a spinner with the caption "is this normal".

So you open the codebase, and within ninety seconds you have three theories, two refactors planned, and a strong urge to swap the JSON library.

Stop.

Put the keyboard down.

The rule before all the other rules

Optimization is not step one. Optimization is step four, after measurement, after confirmation, and after you have found something that is actually slow.

Every optimization in this post buys you speed with complexity.

Cache invalidation, connection pool tuning, pagination cursors, async log buffers, none of that is free. You pay for it forever, in every future debugging session.

So the entry fee is a profile.

Load test the endpoint, look at where the time actually goes, and only then pick a technique from the list below.

The number of times I have watched somebody spend a week optimizing serialization for an endpoint whose real problem was one unindexed query is not a small number.

Measure. Confirm. Then optimize. In that order, every time.

Okay. Assume you measured. Here are the seven things worth reaching for.

1. Caching, or how to not do the work twice

Caching is the highest leverage trick on this list, because the fastest database query is the one you never send.

The shape is simple. An expensive computation runs once, the result goes into Redis or Memcached, and the next N callers asking the same question get the stored answer.

The catch is that people think caching is a big architectural commitment. It usually is not.

def get_top_products(category: str):
    key = f"top_products:{category}"
    if hit := redis.get(key):
        return json.loads(hit)

    result = db.query_expensive_top_products(category)
    redis.setex(key, 30, json.dumps(result))   # thirty seconds. that is it.
    return result

Look at that TTL. Thirty seconds.

That feels almost insultingly short, and it is exactly the point.

If an endpoint takes 400ms and gets hit 200 times a minute, a thirty second cache removes something like 99% of those database hits, and nobody downstream ever notices data that is half a minute stale.

Short TTLs are underrated because they give you most of the win with almost none of the invalidation pain.

You are not maintaining a cache, you are just refusing to answer the same question 200 times in a row.

Where it gets genuinely hard is when the data must be fresh, and then you are in invalidation territory, which is famously one of the two hard things in computer science.

Start with the boring TTL version. Graduate to invalidation only when the TTL version is provably wrong for your use case.

2. Connection pooling, or stop reintroducing yourself

Opening a database connection is not free.

There is a TCP handshake, usually a TLS handshake, then authentication, then session setup.

You can easily spend more time saying hello to Postgres than you spend querying it.

Connection pooling keeps a set of connections open and hands them out. Your request borrows one, runs its query, and gives it back.

Most frameworks do this by default and you never think about it. Which is fine, right up until the day you go serverless.

Serverless breaks the assumption underneath pooling. Each function instance is its own little process with its own little pool, and the platform will happily spin up 500 of them during a traffic spike.

Now your database, which is configured for maybe 100 connections, is getting introduced to 500 strangers at once.

Postgres in particular does not degrade gracefully here. It forks a process per connection, so connection exhaustion is not a slowdown, it is a wall.

That is the entire reason AWS RDS Proxy exists, along with pgbouncer and friends.

They sit between your ephemeral functions and your very much non-ephemeral database, and multiplex a small pool of real connections across a large number of callers.

flowchart LR
    subgraph Serverless
      F1[Fn instance 1]
      F2[Fn instance 2]
      F3[Fn instance ...500]
    end

    F1 --> P[Connection Proxy]
    F2 --> P
    F3 --> P
    P -->|small, reused pool| DB[(Postgres)]

    F1 -.->|without a proxy| DB
    F2 -.->|500 handshakes| DB
    F3 -.->|database says no| DB

    classDef fn fill:#6ea8ff,stroke:#2f5fb8,color:#1a1a1a
    classDef proxy fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef db fill:#ff9a5c,stroke:#c0632c,color:#1a1a1a

    class F1,F2,F3 fn
    class P proxy
    class DB db

3. The N+1 query, the bug that looks like clean code

This is my favourite one, because the N+1 query is almost always the result of writing nicer code.

You have posts. Each post has comments. So you write the obvious thing:

posts = db.query("SELECT * FROM posts LIMIT 20")

for post in posts:
    # one extra round trip. every. single. time.
    post.comments = db.query("SELECT * FROM comments WHERE post_id = %s", post.id)

That is 21 queries to render one page. Twenty of them are identical in shape and differ only in an integer.

On your laptop the database is a localhost away, each query costs 0.2ms, and the whole loop finishes before you can blink.

In production the database is across a network, each round trip costs 3ms, and you have just spent 60ms doing nothing but waiting.

Scale that page to 200 posts and you have an endpoint that is somehow slow without a single slow query in the logs.

That is what makes N+1 so nasty. Every individual query looks fine. The slow query log has nothing to say. Only the count is wrong.

The fix is to stop asking one at a time:

posts = db.query("SELECT * FROM posts LIMIT 20")
ids   = [p.id for p in posts]

rows = db.query("SELECT * FROM comments WHERE post_id = ANY(%s)", ids)

by_post = defaultdict(list)
for row in rows:
    by_post[row.post_id].append(row)

for post in posts:
    post.comments = by_post[post.id]

Two queries. Constant, regardless of how many posts you fetch.

If you are on an ORM, this is what select_related and prefetch_related in Django, joinedload in SQLAlchemy, and include in Prisma exist for.

The tooling is there, it is just off by default, because the ORM cannot know whether you wanted the related rows.

The single most useful habit here is to log your query count per request in development. An endpoint that fires 47 queries will tell on itself immediately.

4. Pagination, because nobody is reading 40,000 rows

Somewhere in every codebase there is an endpoint that started life returning 12 records and now returns 40,000, because the table grew and nobody revisited the handler.

The database has to fetch it. Your app has to serialize it. The network has to ship it.

The client has to parse it, and then render precisely the first twenty of them.

Pagination is the fix and everyone knows it. What everyone does not know is that LIMIT 20 OFFSET 100000 is not actually fast.

OFFSET does not skip work. The database still walks all 100,000 rows and throws them away before handing you twenty.

Deep pages get linearly slower, and your "optimization" quietly becomes the new bottleneck.

Cursor based pagination avoids this by asking the question differently. Instead of "give me page 5000", you ask "give me the twenty rows after this one":

-- offset:  gets slower the deeper you go
SELECT * FROM events ORDER BY id LIMIT 20 OFFSET 100000;

-- cursor:  uses the index, same cost on page 1 and page 5000
SELECT * FROM events WHERE id > 100000 ORDER BY id LIMIT 20;

The second one is an index seek. It costs the same at any depth.

The tradeoff is that you lose random access to page numbers, which is why offset pagination survives in admin panels and cursor pagination is what you find in Stripe's API and every infinite scroll feed you have ever used.

5. Serialization, the tax you pay per response

Once the data is in memory, something has to turn it into JSON, and that something is running on your CPU for every single response.

For small payloads this is noise. For an endpoint returning a few thousand objects, serialization can genuinely become the dominant cost, and the profiler will point right at it.

The good news is that this is the cheapest fix on the entire list, because it is usually a library swap. In Python, orjson is meaningfully faster than the standard library json.

In Node, the JSON serializer is native but schema based approaches like fast-json-stringify beat it by knowing the shape in advance. Serializers that get told the schema upfront can skip all the runtime type sniffing.

But please, actually profile first. Swapping serializers on an endpoint that spends 95% of its time in the database is a lovely way to spend an afternoon achieving nothing.

6. Compression, the one your CDN can do for you

JSON compresses beautifully, because JSON is mostly repeated key names and whitespace. Compression ratios of 5x to 10x on API responses are completely normal.

That is 5x to 10x less data crossing the network, which matters enormously for anyone on mobile, and matters for everyone once payloads get big.

gzip is the safe default that every client on earth supports. Brotli generally compresses smaller at comparable speed and is supported by every modern browser, so it is worth enabling where you can.

Two things to keep in mind.

Compression costs CPU. You are trading processor time for network time, which is nearly always a good trade, but it is still a trade.

Small responses can genuinely come out slower once you add the compression overhead, so most servers have a minimum size threshold, and you should leave it on.

And you very likely should not be doing this yourself. Cloudflare, Fastly and friends will compress at the edge for you, which moves the CPU cost off your servers entirely and applies it uniformly to everything you serve.

If you are already behind a CDN, this optimization is a checkbox.

7. Asynchronous logging, for when microseconds matter

This one is last for a reason. Most services should not care.

But in a high throughput path, writing a log line is a syscall, and if that write is synchronous and blocking, your request thread is sitting there waiting on a disk or a network socket while doing nothing useful.

Async logging fixes this by making the request thread's job trivial. It drops the log entry into an in memory ring buffer and moves on immediately. A separate thread drains the buffer and does the actual writing.

The request path goes from "wait for the write" to "append to a queue".

flowchart LR
    R[Request thread] -->|append, microseconds| B[In-memory buffer]
    B --> W[Logger thread]
    W --> D[(Disk / log service)]

    R --> RESP[Response sent]

    C{App crashes<br/>before flush?}
    B -.-> C
    C -->|yes| L[Buffered logs lost]
    C -->|no| D

    classDef thread fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef buf fill:#6ea8ff,stroke:#2f5fb8,color:#1a1a1a
    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c0632c,color:#1a1a1a

    class R,W thread
    class B,RESP buf
    class C decision
    class L,D bad

That dotted branch is the whole tradeoff, and you should stare at it before enabling this.

Anything sitting in the buffer when the process dies is gone. Which means the logs describing the crash are exactly the logs most likely to be lost.

That is a genuinely bad trade for audit logs, payment records, or anything you would need in an incident review.

It is a perfectly fine trade for high volume access logs where losing the last few hundred lines costs you nothing.

Pick per log stream, not per application.

The actual takeaway

Look at the seven again and notice how they cluster.

Caching, pooling and N+1 are all about not talking to the database, whether by skipping the question, skipping the handshake, or asking once instead of twenty times.

Pagination, serialization and compression are all about moving less data, at the query, at the CPU, and on the wire.

Async logging is about getting out of the request path, which is the same idea as background jobs, applied to something small.

None of them are exotic. All of them are boring, well understood, and sitting one library call away.

The hard part was never knowing the techniques. The hard part is having the discipline to find out which one your endpoint actually needs, instead of applying all seven and calling it architecture.

Profile first. Fix the thing the profile points at. Then go do something more interesting.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

SSH, Actually Explained: Handshakes, Keys, and the Tunnel Trick

Athreya aka Maneshwar — Mon, 07 Sep 2026 17:17:38 +0000

You type ssh ubuntu@1.2.3.4, you get a prompt, and a server in another continent starts obeying your keyboard.

Most of us learned that incantation on day one and never looked under it again.

Which is fair. It works. It has worked for thirty years. Nothing about it demands your attention.

But SSH is doing something genuinely clever in the half second before that prompt appears, and once you have seen it, a whole category of confusing errors stops being confusing.

So let's open it up.

The thing SSH replaced

Before SSH there was telnet, and telnet had exactly one flaw.

It sent everything in plain text. Your username, your password, every command, every byte of output.

On a shared network that is not a subtle problem. Anyone sitting between you and the server was reading your session like a group chat they had been added to.

Telnet is a postcard. SSH is a locked briefcase.

The postman is the same postman in both cases. That is the entire point.

Telnet is still installed on your machine, by the way. It survives as a debugging tool, because telnet host 443 is a quick way to ask "is this port even open".

As a login protocol it is dead, and it deserved it.

The handshake, which is the actually interesting part

Here is the problem SSH has to solve in its first few milliseconds.

Two machines that have never met need to agree on a secret encryption key, while shouting at each other across a network where everything they say is being recorded.

That sounds impossible. It is not, and the reason is the neatest trick in applied cryptography.

SSH uses two kinds of encryption, not one, and people conflate them constantly.

Asymmetric crypto sets up the conversation. It is slow, it involves public and private key pairs, and it is used only for the opening handshake.

Symmetric crypto carries the conversation. One shared key, fast, and it does all the actual work of encrypting your keystrokes.

The handshake exists purely to get both sides holding the same symmetric key.

The move is called a Diffie-Hellman key exchange, and it goes like this.

Each side generates a private value and derives a public value from it.

They swap public values in the clear, where anyone can see them.

Then each side combines its own private value with the other side's public value, and the maths works out such that both arrive at the same number.

An observer who saw both public values cannot get there. They are missing either private half, and going backwards from a public value to a private one is the hard problem the whole thing rests on.

So the session key is never transmitted. It is independently derived, twice, in two different countries.

That key is also ephemeral. It exists for this session and is thrown away when you disconnect. Record the traffic today, steal the server's private key next year, and you still cannot decrypt what you captured. That property is called forward secrecy and it is worth knowing the name of.

Want to actually watch this happen? SSH will narrate it.

$ ssh -vv ubuntu@example.com
debug1: SSH2_MSG_KEXINIT sent
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: Server host key: ssh-ed25519 SHA256:qN8Xb2...
debug1: SSH2_MSG_NEWKEYS sent
debug1: Authenticating to example.com:22 as 'ubuntu'

NEWKEYS is the moment the tunnel goes live. Notice that authentication happens on the line after it.

That ordering matters more than it looks.

The encryption is set up first, and only then does SSH ask who you are.

Your password, if you use one, is already inside the tunnel by the time you type it.

The host key, and that warning you always accept

There is a gap in the story above.

Key exchange gives you a secure channel to somebody. It does not prove that somebody is the server you meant to reach.

That is what the server's host key is for, and it is why the first connection asks you this:

The authenticity of host 'example.com' can't be established.
ED25519 key fingerprint is SHA256:qN8Xb2fV+3Kx9...
Are you sure you want to continue connecting (yes/no)?

You have typed yes there a thousand times without reading it. I have too.

What you are being asked is whether that fingerprint belongs to the machine you think you are talking to. SSH cannot know. There is no certificate authority here, unlike the web.

So it does the next best thing. It writes the fingerprint into ~/.ssh/known_hosts and screams if it ever changes.

That scream is the REMOTE HOST IDENTIFICATION HAS CHANGED block, and it is not being dramatic for fun.

Either the server was genuinely rebuilt, or someone is sitting in the middle pretending to be it.

Nine times out of ten it is a rebuilt server. The tenth time is the reason for the warning.

Keys beat passwords, and here is why

Once the tunnel is up, SSH still needs to know who you are.

You can use a password. You should not.

The better way is a key pair you generate once, on your own machine.

ssh-keygen -t ed25519 -C "laptop"
# ~/.ssh/id_ed25519       <- private. never leaves this machine.
# ~/.ssh/id_ed25519.pub   <- public. paste it anywhere.

ssh-copy-id ubuntu@example.com   # appends the .pub to the server's authorized_keys

Use ed25519, not RSA. It is smaller, faster, and has fewer ways to configure it badly. RSA is still fine at 4096 bits, but there is no reason to pick it for a new key.

The login is a challenge and response.

You tell the server which public key you are claiming. The server checks authorized_keys, finds it, and sends back a random chunk of data.

Your client signs that data with the private key. The server verifies the signature against the public key it already had.

Nothing secret ever crosses the network. Not on the first login, not on the thousandth.

Compare that to a password, which crosses the wire on every single login, lives in someone's head, is probably reused, and is short enough to guess. A 256-bit key is not short enough to guess. That is not a marginal improvement, it is a different category.

This is also why keys are the only sane option for automation. A CI pipeline cannot type a password, but it can hold a key.

You have already been doing this

Two places where this is already in your muscle memory.

EC2. When you launch an instance, AWS puts your public key into the image and hands you the .pem, which is the private half. That is why ssh -i mykey.pem ec2-user@1.2.3.4 works with no password. AWS never had your private key and cannot recover it for you, which is stated plainly in their docs.

GitHub. git push over SSH is the exact same challenge and response. You put your public key in your account settings, and every push signs a challenge. GitHub removed password authentication for Git entirely in 2021, so it is keys or a token, nothing else.

A private key in a git repo is a server someone else owns now. GitHub scans public repos for exactly this, and bots scan them faster.

chmod 600 your keys, keep them out of repos, and put a passphrase on them so a stolen laptop is not a stolen server.

Local port forwarding, the underrated one

Here is the SSH feature that feels like cheating the first time it works.

You need to reach a MySQL database in a private subnet. It has no public IP. It has no route from the internet, on purpose, because it is a database.

But there is a bastion host in front of that network. One small public machine whose entire job is to be the single door, with port 22 open and nothing else.

You can SSH into the bastion. So you can reach the database.

ssh -L 3306:db.internal:3306 ec2-user@bastion.example.com

Read the -L argument as three parts: the local port you want to open, then the host and port to reach from the bastion's point of view.

Now mysql -h 127.0.0.1 -P 3306 on your laptop talks to that private database.

Your MySQL client believes it is connected to something local. It has no idea a tunnel exists. Every byte goes through the encrypted SSH session and comes out on the far side of the firewall.

The same trick works for an internal dashboard, a Redis instance, an admin panel that should never be public, or a staging service someone forgot to expose.

Two directions worth knowing, since the flags are easy to mix up:

-L brings a remote service to your local machine. This is the common one.
-R pushes a local service out to the remote machine. Useful for showing someone your dev server, and also the flag you should audit on production boxes, since it lets an insider open a path inward.

If you find yourself typing long tunnel commands daily, put them in ~/.ssh/config and forget them:

Host db-tunnel
    HostName bastion.example.com
    User ec2-user
    IdentityFile ~/.ssh/prod.pem
    LocalForward 3306 db.internal:3306

Then it is just ssh db-tunnel. Your fingers will thank you.

When it refuses to let you in

Permission denied (publickey) is the least helpful error message in networking, because it covers about six different mistakes.

Here is the order I actually check them in:

flowchart TD
    A[Permission denied publickey] --> B{Did you pass the right key?}
    B -->|No| K[ssh -i mykey.pem or add to ~/.ssh/config]
    B -->|Yes| C{chmod 600 on the key file?}
    C -->|No| P[Fix permissions, SSH ignores world-readable keys]
    C -->|Yes| D{Public key in authorized_keys on the server?}
    D -->|No| U[ssh-copy-id, or paste it in]
    D -->|Yes| E{Right username for the image?}
    E -->|No| N[ec2-user, ubuntu, admin, not root]
    E -->|Yes| F{Home dir or .ssh perms too open?}
    F -->|Yes| H[chmod 700 ~/.ssh, 755 the home dir]
    F -->|No| L[Read the server log: journalctl -u sshd]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef fix fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef last fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a

    class B,C,D,E,F decision
    class A start
    class K,P,U,N,H fix
    class L last

The permissions one catches everybody at least once. SSH will silently ignore a private key that other users can read, which is protective and infuriating in equal measure.

The wrong-username one is the other classic. Amazon Linux wants ec2-user, Ubuntu images want ubuntu, and almost nothing wants root anymore.

When you have exhausted the client side, ssh -vv tells you which keys were offered, and journalctl -u sshd on the server tells you why each one was rejected. Between those two you will find it.

If you run the server side

A short list, because most SSH hardening advice is longer than it needs to be.

Turn off password authentication once your key works. PasswordAuthentication no in sshd_config deletes the entire brute-force attack surface in one line.

Turn off direct root login. PermitRootLogin no, then sudo from a normal user, so the audit log has a name in it.

Do not bother moving off port 22. It stops log noise from untargeted scanners, and nothing else. A real attacker runs a port scan.

Test config changes in a second terminal while your first session is still open. Locking yourself out of a box by reloading a broken sshd_config is a rite of passage best skipped.

What to take away

SSH is not one idea, it is three stacked on each other.

A key exchange that lets two strangers agree on a secret in public, then hands the session to fast symmetric encryption.

An authentication step that proves who you are by signing a challenge, so nothing worth stealing ever crosses the network.

A tunnel that, once you have all that, will carry whatever other protocol you point at it.

That third layer is the one most people never touch, and it is the one that turns SSH from a login tool into a general-purpose way to reach things that are deliberately unreachable.

Go generate an ed25519 key, delete a password login, and forward a port. It is a good afternoon.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

Markov Chain Monte Carlo: the 1953 algorithm hiding under modern AI

Athreya aka Maneshwar — Sun, 06 Sep 2026 04:31:27 +0000

There is an algorithm that got invented in 1953 on a machine with less memory than the favicon of this page.

It is used to forecast the weather.

It is used to fit models of black hole mergers.

It sits under the hood of every serious Bayesian statistics library you have ever imported.

And most working developers have never heard its name.

It is called Markov chain Monte Carlo, MCMC to its friends, and the reason it feels like a black box is that people usually explain it backwards.

They start with detailed balance and ergodicity and stationary distributions, and by the time they get to the part that is actually clever you have closed the tab.

So let's do it forwards.

MCMC is two ideas bolted together, and both of them are simple enough to explain at a bar.

Idea one: you can measure things by throwing stuff at them

Suppose I ask you for the value of pi and take away your calculator.

You could derive it. People have. It is unpleasant.

Or you could draw a square of side 2R, draw a circle of radius R inside it, and start throwing darts at the square while blindfolded.

A dart that lands uniformly in the square has some probability of landing inside the circle, and that probability is just the ratio of the areas.

Circle is pi * R^2. Square is 4 * R^2. Ratio is pi / 4.

So throw a lot of darts, count how many landed inside, multiply by four, and you have pi.

That is the entire Monte Carlo method. It is named after the casino, because the people who invented it at Los Alamos were doing neutron diffusion calculations and needed a codename, and one of them had an uncle who kept borrowing money to gamble in Monte Carlo.

In five lines:

import random

hits = sum(1 for _ in range(10_000_000)
           if random.random()**2 + random.random()**2 <= 1.0)

print(4 * hits / 10_000_000)   # 3.1417...

Nobody in that snippet solved an integral. They counted.

The catch, and this is the catch that the whole rest of the article exists to fix, is that random.random() gave us independent samples for free.

We knew the shape we were sampling from. It was a square. Sampling uniformly from a square is easy.

In every problem you actually care about, you do not know how to sample from the shape. That is the problem.

Idea two: a process that only remembers where it is right now

A Markov chain is a sequence of states where the next state depends only on the current one.

Not on how you got here. Not on the previous forty steps. Just on here.

This is called the Markov property, and the shorthand for it is that the chain is memoryless.

The textbook example is weather, so let's use the textbook example. Three states: rainy, cloudy, sunny. Fixed probabilities for hopping between them.

From rainy there is a 60% chance you go to cloudy and a 40% chance you stay put. From cloudy there is a 50% chance you go to sunny. And so on.

Now run it. Rainy, cloudy, sunny, sunny, cloudy.

Step five consulted step four and nothing else. It has no idea that the run started in the rain.

Here is the property that makes this useful instead of merely cute.

If the chain can reach every state and does not get permanently trapped anywhere, then the fraction of time it spends in each state converges to a fixed set of numbers.

Run it a thousand steps and you might be rainy 30% of the time. Run it a million and it is still 30%. It has settled.

That set of numbers is the stationary distribution, and it does not depend on where you started.

Sit with that for a second, because it is the whole trick.

A Markov chain, left alone, generates samples from a distribution. Not a distribution you chose. Just whatever distribution falls out of the transition rules you happened to write down.

So what if you ran that backwards?

What if you had a distribution you wanted, and you designed the transition rules so that its stationary distribution was exactly that thing?

Then you would have a machine that spits out samples from a distribution you were never able to sample from directly.

That is MCMC. That is the entire idea. Everything else is engineering.

The distribution nobody can compute

Time to be concrete about what "a distribution you cannot sample from" actually means, because otherwise this is all very abstract.

Say you are doing Bayesian inference. You have data, you have a model with some parameters, and you want to know which parameter values are consistent with what you observed.

A non-Bayesian method hands you one best fit number and a standard error.

Bayesian inference hands you a whole distribution over parameters, called the posterior, which tells you both what is likely and how confident you are allowed to be about it.

You get it from Bayes' theorem, which is four symbols and one enormous problem.

The numerator is fine. The likelihood is "how well do these parameters explain my data", which you can evaluate.

The prior is "what did I believe before I saw the data", which you wrote down yourself.

The denominator is where it falls apart.

It is called the evidence, and it is an integral over every possible combination of every parameter.

It exists purely to make the whole thing sum to one.

With one parameter, grid it at 100 points, evaluate 100 times, done before your coffee lands.

With five parameters, that is ten billion evaluations.

With ten parameters, which is a small model by any modern standard, you are at 10^20 and the sun has opinions about your timeline.

This is the curse of dimensionality, and it is not a performance problem you can engineer around. Grid methods die here. Every time.

So you are stuck holding a distribution that you want to draw samples from, and you cannot even evaluate it, because evaluating it requires a normalising constant you cannot compute.

Which sounds terminal.

The cancellation that saves everything

Here is the move.

MCMC never asks "what is the posterior probability at this point".

It only ever asks "is this new point better or worse than the one I am standing on, and by how much".

That is a ratio. And in that ratio, the evidence appears on the top and on the bottom.

So it cancels.

You never compute the impossible thing. You just arrange never to need it.

The consequence is that you only need the posterior up to a constant, which is just likelihood times prior, which you can always evaluate.

That is the load-bearing insight of the entire field.

Metropolis-Hastings, the whole thing

The oldest MCMC algorithm is Metropolis et al., 1953, later generalised by Hastings in 1970. It is short enough to hold in your head.

You are standing at some parameter value. You want to take a step.

Propose. Draw a candidate near where you are, usually from a Gaussian centred on your current position.

Score. Compute R = p(proposed) / p(current), using the unnormalised posterior, because that is all you have and all you need.

Decide. If R >= 1, the new spot is better. Move there. Always.

If R < 1, the new spot is worse. Move there anyway, with probability R.

That last line is the one people skip past, and it is the one that makes the algorithm work.

If you only ever accepted uphill moves, you would have written a hill climber.

It would sprint to the nearest peak, sit on it, and report that peak as the answer with total confidence, having never seen the rest of the distribution.

The occasional deliberately bad move is what lets the chain roll down one hill and find another. It is what turns a greedy optimiser into a sampler.

And the acceptance rule is not arbitrary. It is constructed so that the chain's stationary distribution is exactly the posterior you handed it.

The chain visits high probability regions often and low probability regions rarely, in precisely the right proportion.

So it looks like a random walk, and it kind of is, but it is a random walk with a rigged floor.

Here is the whole algorithm, and I mean the whole algorithm:

import numpy as np

def metropolis(log_post, start, n_steps, step_size):
    x, lp = start, log_post(start)
    chain = []
    for _ in range(n_steps):
        candidate = x + np.random.normal(0, step_size, size=np.shape(x))
        lp_new = log_post(candidate)
        # log space, so the ratio is a subtraction and nothing overflows
        if np.log(np.random.rand()) < lp_new - lp:
            x, lp = candidate, lp_new
        chain.append(x)          # note: appended even when we rejected
    return np.array(chain)

Two things in there that trip people up.

We work in log space. Posteriors underflow to zero fast in float64, so the ratio becomes a subtraction of log densities. np.log(rand()) < lp_new - lp is the same rule, numerically survivable.

We append x even on rejection. Staying put is a real outcome. If you only recorded accepted moves you would systematically under-count the sharp peaks, which are exactly the regions where most proposals get rejected.

As a flowchart:

flowchart TD
    A[Pick a starting value] --> B[Propose a new value<br/>from a Gaussian around it]
    B --> C[Compute R = p_new / p_current]
    C --> D{R >= 1?}
    D -- yes, uphill --> E[Accept the move]
    D -- no, downhill --> F{Coin flip lands<br/>under R?}
    F -- yes --> E
    F -- no --> G[Reject, stay put<br/>and record the old value again]
    E --> H[Record the sample]
    G --> H
    H --> I{Still in burn-in?}
    I -- yes --> J[Throw this sample away]
    I -- no --> K[Keep it]
    J --> B
    K --> L{Enough samples?}
    L -- no --> B
    L -- yes --> M[Average the samples<br/>to get anything you want]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c65f22,color:#1a1a1a
    classDef work fill:#6ea8ff,stroke:#2f5fbf,color:#1a1a1a

    class D,F,I,L decision
    class A,M start
    class E,K,H good
    class G,J bad
    class B,C work

The step size is the one knob, and it will bite you

step_size looks like a tuning detail. It is not. It is the difference between a chain that works and a chain that lies to you.

Make the proposals too narrow and almost everything gets accepted, because you barely moved. The chain shuffles across the distribution at a glacial pace and you will need millions of samples to see anything.

Make them too wide and almost everything lands somewhere terrible and gets rejected. The chain stands still for hundreds of steps at a time, and your "10,000 samples" are actually about forty distinct values repeated.

Both failures look like success from the outside. You get your array of 10,000 numbers either way.

The folk rule for the simple random walk version is to aim for an acceptance rate around 20 to 25%, which comes from some genuinely lovely asymptotic work by Roberts, Gelman and Gilks. Print the acceptance rate. Always print the acceptance rate.

This is also why nobody writes the loop above in production. Modern samplers like Stan and PyMC use Hamiltonian Monte Carlo and NUTS, which use gradients of the posterior to propose smart, distant moves instead of blind local wobbles, and tune themselves. The idea is identical. The proposal is just far less stupid.

Burn-in, or: throwing away the work you paid for

You have to start the chain somewhere, and your somewhere is probably wrong.

If your initial guess lands in a region of terrible posterior probability, the chain will spend a while wandering out of the wilderness before it finds the part of parameter space that matters.

Those early samples are real samples, they cost real compute, and they are garbage. They describe your bad guess, not the posterior.

But remember the Markov property. The chain has no memory of where it started. Once it reaches the high probability region, it stays there, and nothing about its future behaviour is contaminated by the trek it took to get there.

So the fix is embarrassingly blunt. Delete the first few hundred or few thousand samples.

That is burn-in. It is not a hack, it is a direct consequence of memorylessness.

The way you check is the trace plot on the left of that diagram. Plot parameter value against iteration. A healthy chain looks like a fuzzy horizontal caterpillar. A chain still climbing in, or drifting, or sitting flat for long stretches, is telling you something and it is not good news.

In practice people run several chains from different starting points and check they all converge to the same place, which is what the R-hat statistic measures.

What you do with a pile of samples

Once you have samples from the posterior, the hard part is over and everything downstream is embarrassingly easy.

Want the mean of a parameter? Average the samples.

Want a 95% credible interval? Sort them and take the middle 95%.

Want the probability that a parameter is bigger than 10? Count how many are, divide by how many you have.

chain = metropolis(log_post, start=0.0, n_steps=50_000, step_size=0.8)
samples = chain[5_000:]                       # drop burn-in

samples.mean()                                # point estimate
np.percentile(samples, [2.5, 97.5])           # 95% credible interval
(samples > 10).mean()                         # P(theta > 10), directly

Every question you can ask about a distribution is an expectation, and every expectation is approximated by an average over samples. That is the Monte Carlo half, quietly doing its job at the end.

Which is where the name finally makes sense. The Markov chain gives you the samples. The Monte Carlo gives you the answers. Neither half works alone.

Where the two halves meet

flowchart TD
    P[A posterior you cannot integrate] --> Q{Can you draw<br/>independent samples?}
    Q -- yes --> MC[Plain Monte Carlo<br/>throw darts, average them]
    Q -- no --> R{Can you at least<br/>evaluate it up to a constant?}
    R -- no --> STUCK[You are genuinely stuck]
    R -- yes --> CHAIN[Build a Markov chain<br/>whose stationary distribution<br/>is that posterior]
    CHAIN --> WALK[Walk it for a long time]
    WALK --> S[Correlated samples that still<br/>average to the right answer]
    MC --> ANS[Means, intervals, probabilities]
    S --> ANS

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef chip fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef accel fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c65f22,color:#1a1a1a

    class Q,R decision
    class P start
    class MC,S,ANS chip
    class CHAIN,WALK accel
    class STUCK bad

One honest caveat, because I do not want to oversell this.

MCMC samples are correlated. Consecutive steps are near each other by construction, so 10,000 MCMC samples carry less information than 10,000 independent ones. The quantity that matters is the effective sample size, and it can be brutally smaller than the number you generated. Every decent library reports it. Look at it.

MCMC also does not automatically work. It converges eventually, and "eventually" is doing real work in that sentence. A posterior with two well separated modes and a deep valley between them can keep a random walk chain trapped on one mode for longer than you are willing to wait, and the chain will look perfectly healthy the whole time. Convergence diagnostics are not paranoia, they are the job.

So why does this matter to you

Because the pattern generalises well past statistics.

You have an object you cannot enumerate. You cannot compute its total. But you can compare two candidates cheaply, and you can take a random step.

That is enough. That is the whole precondition.

Simulated annealing is this. So is a large chunk of statistical physics. So is the PageRank random surfer, which is a Markov chain whose stationary distribution is the ranking. So is every diffusion model generating an image right now, iteratively walking noise toward a distribution it learned rather than one you wrote down.

The 1953 paper was about hard spheres in a box.

SIAM later put the Metropolis algorithm on its list of the top ten algorithms of the twentieth century, next to the FFT and QR decomposition.

So next time someone asks how you do inference over a distribution you cannot compute, you have an answer.

You do not compute it.

You build a chain that walks through it for you, and you let the walking do the arithmetic.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

DEV Community: Athreya aka Maneshwar

GFS Cost More Than 50 Cents a Run. Now Each Run Costs 4 Cents

Cheap RAG in Go with Gemini File Search: no vector DB, two calls, one hosted store

The word "free" was doing a lot of work

What we swapped in

Chunking is where the accuracy actually lives

Dense finds the paraphrase, BM25 finds the name

The reranker that got fired

Two hours of embedding, four laptops

Now that retrieval is free, spend it on drafts

The check that replaced trusting the grounding metadata

What the bill looks like now

What I would steal

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

Cheap RAG in Go with Gemini File Search: no vector DB, two calls, one hosted store

RAG without owning the R

Step zero: getting books into markdown without losing the spaces

The store belongs to the key, not to you

Ingest is a diff, not an upload

Search with the pattern, not the post

Every instruction is a law, every reply cites its laws

Trust, but run the checks

The bug that made me split the database

What the free tier actually gives you

What I would tell you to steal

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

How to Count 100 Billion Things in 12 Kilobytes

The naive fix, and the bill for it

The trick: rare things are evidence of scale

Averaging away the noise

Somebody already built this into your database

When you actually still want the hash set

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

MVC, MVP, MVVM, MVVM-C, VIPER: one pattern wearing five outfits

The one shape underneath all of them

MVC: everything reports to one controller

MVP: the Presenter takes the homework off the View

MVVM: the View stops asking, and starts just knowing

MVVM-C: somebody still has to decide where the app goes next

VIPER: five jobs, five files, no exceptions

So which one do you actually pick

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

The ffmpeg Pipeline Nobody Explains

The pipeline nobody tells you about

The flag that changes everything: -c copy

When you actually need the expensive path

The rest of the toolbox

Why this is worth knowing

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

Seven Patterns That Decide If Your AI App Survives 10,000 Users

Zoom out one level

Pattern 1: the API gateway is your reception desk

Pattern 2: rate limiting decides how much work gets to start

Pattern 3: cache what's actually safe to reuse

Pattern 4: let the work that can wait, wait

Pattern 5: stop one broken dependency from taking the site with it

Pattern 6: spread the load across more than one copy

Pattern 7: autoscaling adds the capacity the rest bought you room to add

Walking one request through the whole thing

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

How Uber Knows Your Driver Is 7 Minutes Away

Step 1: chop the world into segments

Step 2: don't ask Google, ask your drivers

Step 3: the part that actually needs machine learning

The payoff

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

The flag that changes everything: `-c copy`