DEV Community: Sarthak Rawat

Retrieval-Augmented Generation, Explained From First Principles

Sarthak Rawat — Mon, 28 Sep 2026 12:43:52 +0000

Large language models are remarkably good at producing answers. Ask one to explain a difficult idea, summarize a document, write code, or reason through a problem, and it can often do it in seconds.

But there is a fundamental limitation hiding underneath all of that fluency: the model's knowledge is not the same thing as access to the information you need right now.

A model may know a great deal about a subject and still know nothing about your company's latest policy, yesterday's support tickets, this month's internal report, or the newest version of a regulation. It may also produce a confident answer when the information it needs is missing.

Retrieval-Augmented Generation, or RAG, is one of the most practical ways of dealing with that problem.

The basic idea is simple:

Instead of asking the model to remember everything, give it the
relevant information at the moment it needs to answer.

The simplicity of that sentence is deceptive. A production RAG system is a chain of decisions about documents, representations, search, ranking, context, generation, evaluation, security, latency, and cost. Most of the interesting engineering happens in those decisions.

This article builds RAG from the ground up, then follows the system into the problems that appear when a simple prototype becomes a real application.

The problem RAG is trying to solve

Imagine a brilliant consultant who has read an enormous library of books, papers, manuals, and websites. They can reason extremely well, explain difficult ideas, and synthesize information quickly.

There is one catch: they were locked in a room after a particular date.

They have no idea what happened afterward.

There is another problem. When they do not know something, they may still try to give you an answer. Because they are optimized to produce plausible language, the result can sound convincing even when it is wrong.

That is a useful mental model for an LLM.

Now imagine giving the consultant a research assistant. The assistant can search a filing cabinet, find the relevant pages, and put them on the consultant's desk before the consultant answers.

The consultant still does the reasoning. The assistant provides the evidence.

That is the central idea behind RAG.

Retrieval finds relevant external information.
Augmentation puts that information into the model's context.
Generation produces an answer using that context.

The model's reasoning ability and its external knowledge have been separated.

RAG decouples what a model can reason about from what it currently has access to.

That distinction is more important than memorizing a particular RAG framework or vector database.

RAG, fine-tuning, and long context solve different problems

A common mistake is to treat RAG and fine-tuning as competing ways of doing the same thing.

They are not.

RAG is primarily a way to give a model access to information at inference time. Fine-tuning changes the model's learned behavior. A larger context window gives the model more room to receive information, but does not by itself create a retrieval mechanism.

A useful shorthand is:

Fine-tuning teaches the model how to behave; RAG gives the model what to know.

In practice, the techniques can be combined. A specialized model can learn domain vocabulary, formatting, or communication style through fine-tuning while RAG supplies current private knowledge.

Long context changes the tradeoff again. If the complete information needed for a task is small enough to fit comfortably into context, retrieval may be unnecessary. RAG becomes more useful when the corpus is large, changing, private, or expensive to stuff into every prompt.

There is no universal winner. The choice depends on the information, latency, cost, update frequency, and reliability requirements of the application.

The RAG pipeline

At its simplest, RAG can be understood as four stages:

Index → Retrieve → Augment → Generate

The first stage prepares the knowledge base. The other stages happen when a user asks a question.

1. Indexing: prepare the knowledge

Before anyone asks a question, the system needs to turn its source material into something searchable.

A typical indexing pipeline looks like:

Load → Parse → Chunk → Embed → Store

Load

The source might be almost anything:

PDFs
Word documents
web pages
database records
Confluence pages
Slack messages
support tickets
internal manuals
code or technical documentation

The important point is that RAG does not require the knowledge to have originated as neatly structured text.

Parse and clean

Raw documents are rarely ready for retrieval.

A PDF may contain headers and footers repeated on every page. A webpage may contain navigation menus. A table may be extracted in the wrong order. OCR may introduce errors.

If the parser turns a useful document into poor text, every later stage inherits the damage.

This is one of the least glamorous parts of RAG and one of the most consequential:

Garbage in becomes searchable garbage.

Ingestion also has to deal with practical questions such as
deduplication, deletions, document versions, permissions, and what happens when the embedding model changes. These are not separate from RAG quality; they are part of it.

Chunking

A 200-page document is rarely a useful retrieval unit.

Instead, it is divided into smaller passages called chunks.

The goal is to find a useful balance.

Chunks that are too small can lose the context needed to understand a statement. Chunks that are too large contain too much unrelated information and become less precise retrieval units.

A common starting point is a few hundred tokens with some overlap, but there is no universally correct chunk size. A legal contract, a support ticket, a code file, and a research paper may all want different structures.

Overlap exists for a simple reason: boundaries are artificial.

If a sentence begins at the end of one chunk and finishes at the
beginning of another, retrieving either chunk alone could leave the model with an incomplete idea.

More advanced approaches include parent-child retrieval, where small child chunks are indexed for precise retrieval but a larger parent passage is supplied to the model after a match is found.

The important principle is:

Chunk for retrieval, not merely for storage.

Embeddings

Once chunks exist, the system needs a way to represent their meaning numerically.

An embedding is a vector: a sequence of numbers produced by an embedding model. Text with related meanings should generally occupy nearby regions of the embedding space.

For example, a user might ask:

"Can I work from home?"

while the document says:

"Employees may work remotely up to three days per week."

The wording is different, but the meaning is related. Dense semantic retrieval can recognize that relationship.

The embedding model is not the same thing as the generative LLM. The embedding model creates representations for search. The LLM reads context and generates language.

One important engineering rule follows from this:

The query and the indexed documents need compatible
representations.

If documents were embedded using one model and queries are embedded using an incompatible model, the vectors are not necessarily meaningful to compare.

Dense and sparse representations

Dense embeddings are excellent at semantic similarity, but they are not the only useful representation.

Sparse retrieval methods such as BM25 are strongly tied to terms
appearing in the text. That makes them particularly useful for exact names, product codes, error messages, rare technical identifiers, and other cases where the literal words matter.

Dense and sparse retrieval therefore have complementary strengths.

That is why production systems often combine them.

What happens inside vector search?

Once chunks have embeddings, they need to live somewhere that can search them efficiently.

A vector database or vector index stores the vector alongside the
original text and useful metadata such as:

source
author
department
date
document version
access permissions

When the user asks a question, the query is embedded into the same representation space and compared with stored vectors.

A common similarity measure is cosine similarity, which focuses on the angle between vectors. Dot product is another common option, while L2 distance measures geometric distance directly. Normalization matters because it changes the relationship between these measures.

The important idea is less about memorizing one formula than
understanding what the system is doing:

It is trying to find stored representations that are close to the query under the chosen similarity function.

Searching every vector exactly becomes expensive as the corpus grows.
Approximate nearest-neighbor (ANN) methods trade some exactness for much faster search.

The important word is approximate. Exact nearest-neighbor search asks the system to compare the query against every candidate and identify the true closest vectors. That can become too expensive at large scale.
ANN methods instead organize the search space so that the system can explore a much smaller set of promising candidates.

HNSW: navigating a graph

HNSW (Hierarchical Navigable Small World) is a popular graph-based ANN method.

A useful intuition is a city map with several levels. The upper levels contain a small number of long connections that let you move quickly across the map. Lower levels contain increasingly detailed local connections.

A query starts at a higher level, moves toward promising regions, then descends through the hierarchy until it is navigating the detailed neighborhood containing the closest candidates.

The result is that the system does not need to examine every stored vector.

HNSW also exposes an important engineering tradeoff through parameters such as ef_search. A larger search effort means the algorithm explores more candidate nodes. That can improve recall, but it also increases search work and therefore latency. A smaller value can make search faster while increasing the chance that a genuinely good neighbor is missed.

So even within the same vector database, ANN quality is not simply “on” or “off.” You are choosing where to sit on a speed-versus-recall curve.

IVF: search the relevant regions

Another common approach is inverted file indexing (IVF).

Instead of treating the entire vector space as one enormous search region, IVF partitions vectors into clusters. During indexing, each vector is assigned to an appropriate cluster.

At query time, the system first determines which clusters are closest to the query, then searches only those regions.

The important tuning concept is nprobe: how many clusters should the query inspect?

A low nprobe means less work and lower latency, but it can miss
relevant vectors that live in a cluster the search never examined. A higher nprobe searches more regions and can improve recall at the cost of additional computation.

HNSW and IVF therefore use different structures to make the same broad tradeoff: avoid exhaustive search while keeping the probability of missing useful neighbors acceptably low.

Product quantization: make the vectors smaller

There is another problem at scale: memory.

A large corpus may contain hundreds of millions of vectors, each with hundreds or thousands of dimensions. Keeping every vector in full precision can become expensive.

Product Quantization (PQ) compresses vectors into smaller
representations. Instead of storing every original value directly, the system represents portions of the vector using compact codes.

The benefit is substantially lower memory usage and potentially faster search. The cost is that compression introduces approximation, which can reduce retrieval accuracy.

This is another example of the same RAG engineering principle: you are trading one resource for another.

Why ANN recall becomes an engineering concern

It is tempting to describe ANN as simply “less accurate search.” That is too simplistic.

The real issue is that as the corpus grows, maintaining very high recall under a fixed latency and memory budget becomes harder. The system has more possible neighbors to consider, more index structure to maintain, and more pressure to limit how much work each query performs.

That is why ANN systems expose tuning parameters and why retrieval should be evaluated empirically.

In practice, you care about questions such as:

How much recall@k do we lose at our latency target?
What happens when the corpus grows tenfold?
How much memory does the index require?
Which parameters improve recall without making latency unacceptable?
Does compression change the ranking enough to affect downstream answer quality?

The useful mental model is therefore not “ANN is an approximation.”

It is:

ANN is a controlled tradeoff between search cost and the probability of finding the best neighbors.

Other ANN approaches include inverted-file methods and vector
compression techniques such as product quantization. The specific choice depends on corpus size, latency requirements, available memory, and the recall you need.

Metadata filtering is another important capability. You may want
semantic similarity only among documents from the legal department, or only among documents newer than a certain date, or only among documents the current user is allowed to see.

That last requirement becomes critical in multi-tenant and
permission-sensitive systems. A retrieval system that finds the right answer but exposes information the user is not authorized to see is still a broken system.

Retrieval: why "just use vector search" is not enough

A basic RAG prototype might embed the query, perform a similarity
search, take the top five chunks, and send them to the LLM.

Sometimes that works beautifully.

Sometimes it fails in ways that are difficult to diagnose.

There are several recurring reasons.

Vocabulary mismatch

A user may say "heart attack" while the document says "myocardial
infarction."

Semantic retrieval can help bridge that gap, but no retrieval method is perfect.

Exact terms matter too

Now imagine the query contains a product ID such as XR-4729, a legal clause number, or a specific error code.

Semantic similarity is not necessarily the best tool for that.

This is where sparse keyword retrieval can be extremely valuable.

Query and document lengths differ

A user query might be six words while the retrieved chunk is several hundred words. Comparing the two is not exactly the same problem as comparing two similarly sized documents.

Redundancy

The top ten results may all say essentially the same thing.

Sending ten versions of one idea to the LLM does not necessarily make the answer better.

Hybrid retrieval: combine complementary searches

A common production baseline is hybrid search.

Run a dense semantic search and a sparse lexical search in parallel, then combine their rankings.

BM25 is a classic sparse retrieval method. It rewards useful term
matches while accounting for how common a term is across the collection.

Dense retrieval contributes semantic matching.

BM25 contributes exact lexical matching.

The results can be combined using a ranking method such as Reciprocal Rank Fusion (RRF), where a document receives contributions based on where it appeared in each ranked list.

The important insight is not the formula itself. It is that two
imperfect retrieval mechanisms can complement one another.

Query rewriting and decomposition

The user's raw question is not always the best search query.

A conversational question such as:

"What about the policy for contractors?"

may depend on several earlier turns.

A query rewriting step can turn conversational language into a
standalone retrieval query.

More complex questions can also be decomposed.

Suppose someone asks:

"How did our Q3 revenue compare with the previous quarter, and what were the main reasons for the change?"

That may require separate retrieval for the numbers and for the
explanation.

Instead of treating the whole thing as one search query, the system can create subqueries, retrieve evidence for each, and combine the results.

This adds complexity and sometimes extra model calls, so it should be used when the query actually benefits from it.

HyDE and multi-query retrieval

HyDE, or Hypothetical Document Embeddings, takes another approach.

Instead of embedding the user's short question directly, the system first asks an LLM to generate a hypothetical answer or document. That hypothetical text is then embedded and used for retrieval.

The hypothetical text does not have to be factually correct. Its purpose is to provide a richer representation of what a relevant document might look like.

The tradeoff is straightforward: another model call means additional latency and cost.

Multi-query retrieval is simpler conceptually. The system generates several alternative formulations of the same question, retrieves for each, merges and deduplicates the results, and then ranks the candidates.

These techniques are useful when the original query is a poor
representation of the information the system actually needs.

They are not mandatory ingredients of every RAG system.

Reranking: retrieve broadly, then judge carefully

One of the most useful patterns in practical RAG is to separate
candidate generation from precise ranking.

The first retrieval stage should be fast. It might return 20, 30, or 50 candidates.

Then a more expensive cross-encoder reranker examines the query and each candidate together and scores how well they match.

This works because the two stages have different jobs.

A bi-encoder or vector search system is efficient because the document representations can be computed ahead of time.

A cross-encoder can make a more detailed query-document comparison, but doing that against millions of chunks would be too expensive.

So the architecture becomes:

large corpus → fast retrieval → small candidate set → expensive
reranking → tiny final context

This is one of the most important patterns to understand in RAG
engineering.

Diversity matters too

Sometimes the five highest-ranked chunks all repeat the same
information.

Maximal Marginal Relevance (MMR) provides a way to balance relevance with diversity.

Conceptually, MMR rewards a chunk for being relevant to the query while penalizing it for being too similar to chunks already selected.

The result is a final context containing several complementary pieces of evidence instead of five near-duplicates.

The broader lesson is important:

Retrieval is not about finding the most similar text. It is about constructing the most useful evidence set.

The generation stage: context is evidence, not decoration

Once the system has retrieved its evidence, it needs to assemble the prompt that the LLM will see.

A useful RAG prompt typically contains:

clear system instructions
labelled source chunks
grounding rules
the user's question
a defined output format when necessary

The structure matters because the model is now being asked to reason over evidence.

A good instruction might say:

Answer using only the provided context. If the answer is not supported by the context, say that the information is insufficient. Do not speculate beyond the sources.

That does not magically make hallucination impossible. But it gives the model a clear operating rule.

More context is not automatically better

It is tempting to think:

"If five chunks are useful, twenty must be better."

Usually, it is not that simple.

Every additional chunk increases token usage and latency. Irrelevant chunks can introduce noise. Conflicting chunks can make the model uncertain. And long contexts can create attention problems.

This is related to the lost-in-the-middle phenomenon: when relevant information is buried among a long sequence of context, models may use information at the beginning and end more effectively than information buried in the middle.

A practical strategy is therefore:

Retrieve more, send less.

Retrieve a broad candidate set. Rerank it. Select a small, high-quality context.

Citations

One of RAG's practical advantages is that the system knows which sources it retrieved.

Citations can be rendered inline:

Employees may work remotely up to three days per week [Source 1].

Or as a source list:

Sources: HR Policy, page 4; Manager Handbook, page 12.

In production, structured citation data can be even more useful. The system can return an answer together with source IDs, pages, claims, and links so the interface can make evidence clickable and auditable.

What happens when the answer isn't in the documents?

A trustworthy RAG system needs an explicit answer to this question.

If the retriever finds nothing relevant, the model should not feel obligated to invent an answer.

A simple design is to combine grounding instructions with retrieval thresholds. If none of the retrieved candidates passes an appropriate relevance threshold, the system can decline to answer from the knowledge base.

The exact threshold is application-specific. A score from one embedding system cannot automatically be treated as a universal measure of relevance.

The deeper principle is more important:

A good RAG system must have a graceful failure mode.

"I don't have enough information in the retrieved sources" is often much better than a confident fabrication.

Conversational RAG: the query is not always the query

Chat makes retrieval harder.

A user might say:

"What's the refund policy?"

Then:

"What about digital products?"

Then:

"And if I used a gift card?"

The final message is not a complete retrieval query.

A common solution is query condensation: use the conversation
history to rewrite the latest message into a standalone question before retrieval.

For example:

"What is the refund policy for digital products purchased with a gift
card?"

Now the retrieval system has the information it needs to search
effectively.

This adds another processing step, but it solves a fundamental mismatch between how humans converse and how retrieval systems search.

When RAG gives a bad answer: debug the pipeline, not just the model

One of the most useful ways to think about RAG is that a bad answer does not necessarily mean the LLM is the problem.

A failure can happen at several layers:

Ingestion failure\
The relevant information was parsed incorrectly, omitted, duplicated, or indexed incorrectly.

Chunking failure\
The information exists, but it was divided into retrieval units that destroyed the useful context.

Retrieval failure\
The right information exists in the index but was never retrieved.

Ranking failure\
The right information was retrieved but buried below less useful
candidates.

Context assembly failure\
The correct evidence was found but was poorly ordered, excessively long, or mixed with conflicting material.

Generation failure\
The model received useful evidence but failed to follow it or answer the question correctly.

This gives you a practical debugging order:

Check the source → check the retrieved evidence → check the
ranking → check the assembled context → check the generation.

Do not immediately swap the LLM when the real problem is retrieval.

How do you know whether a RAG system actually works?

A RAG system can produce answers that sound excellent while failing underneath.

Evaluation needs to separate the pipeline into understandable
relationships.

The RAG triad

Three questions form a useful mental model:

1. Context relevance

Did the retriever find information that was actually related to the question?

2. Faithfulness

Are the claims in the generated answer supported by the retrieved
context?

3. Answer relevance

Does the answer actually address what the user asked?

These dimensions catch different failures.

A system can retrieve excellent evidence and still hallucinate.

It can produce a faithful answer that does not answer the user's
question.

It can generate a relevant answer because the model already knew the topic, even though retrieval failed.

Retrieval metrics

The RAG triad is useful for end-to-end reasoning, but retrieval also benefits from traditional information-retrieval metrics.

Recall@k asks whether the relevant evidence appears somewhere in the top k results.

MRR (Mean Reciprocal Rank) cares about how early the first relevant result appears.

nDCG evaluates ranked results while giving more credit to highly relevant items near the top.

These metrics answer a different question from faithfulness. They tell you whether the retrieval system is finding and ordering useful
evidence.

RAGAS and LLM-as-judge

Frameworks such as RAGAS operationalize several RAG evaluation concepts, often using an LLM as a judge.

One useful technique is to break an answer into atomic claims and verify those claims against the retrieved context.

For example:

Claim 1 → supported\
Claim 2 → supported\
Claim 3 → unsupported

That gives a more meaningful faithfulness assessment than asking whether an answer "looks good."

LLM-as-judge is powerful because it scales, but it is not an
unquestionable authority. Judges can have biases, prefer verbose
answers, or make mistakes of their own.

A strong evaluation system therefore combines automated evaluation with carefully constructed golden datasets and periodic human validation.

A practical test set can combine:

expert-written questions
synthetic questions generated from the corpus
sampled real production queries

The important part is establishing a baseline and using it to detect regressions when the RAG pipeline changes.

When basic RAG isn't enough

Basic RAG makes a strong assumption:

Retrieve something, use it, and generate an answer.

That assumption breaks in several ways.

Sometimes the question does not require retrieval.
Sometimes the retrieved information is poor.
Sometimes answering requires relationships across many documents.

Sometimes the system needs to decide what tool or retrieval strategy to use next.

Several advanced RAG patterns address these problems.

Self-RAG: should I retrieve, and did retrieval help?

Basic RAG assumes retrieval should happen for every query.

That is wasteful for questions that the model can answer directly, and it can actively hurt when retrieval returns poor evidence.

Self-RAG changes that assumption by giving the model a role in deciding when retrieval is needed and in evaluating whether retrieved information is useful.

The original approach uses special reflection signals generated by a model that has been trained for this behavior. In other words, the reflection mechanism is part of the model's learned generation process; it is not simply a separate classifier bolted onto an ordinary LLM.

Conceptually, the loop looks like:

question → decide whether retrieval is useful → retrieve → assess the evidence → generate

This makes Self-RAG attractive for systems that receive a mixture of questions: some needing external knowledge and others that do not.

The tradeoff is important, though. The approach requires a model capable of the specialized reflection behavior, which makes it more involved than simply adding another retrieval step to an existing RAG pipeline.

Corrective RAG: don't trust the retriever blindly

A standard RAG pipeline can make a dangerous assumption:

If the retriever returned something, it must be useful.

Corrective RAG (CRAG) explicitly challenges that assumption.

CRAG introduces a quality assessment around retrieved documents. If the retrieved material is judged weak, the system can take corrective action instead of blindly passing the results to the generator.

One part of the original CRAG idea is knowledge refinement. Retrieved chunks can be broken into smaller units, irrelevant material can be discarded, and the remaining high-signal information can be recombined before generation.

The conceptual shift is simple:

Basic RAG: retrieve → trust → generate

CRAG: retrieve → evaluate → refine/correct → generate

This is particularly useful when document quality is uneven or the retrieval system cannot be assumed to return reliable evidence every time.

GraphRAG: when relationships matter

Vector search answers a question like:

“Which passages are most similar to this query?”

That is powerful, but some questions are fundamentally about
relationships.

Imagine a question that requires connecting an employee to a department, that department to a project, and that project to a supplier. The useful information may be spread across many documents and may not appear as one semantically similar passage.

GraphRAG addresses this by representing entities and relationships explicitly.

A typical graph-oriented pipeline can involve:

extracting entities from source documents
identifying relationships between those entities
constructing a knowledge graph
detecting communities or groups of closely connected entities
generating summaries of those communities
retrieving through graph structure at query time

This creates two useful modes of reasoning.

Local search focuses on a particular entity and its surrounding neighborhood.

Global search can reason over broader themes represented by
community-level summaries.

GraphRAG therefore becomes attractive when the question depends on connections rather than simply semantic similarity: organizational structures, citation networks, supply chains, interconnected technical systems, and similar domains.

The cost is substantial. Building and maintaining the graph can require many extraction steps and additional LLM calls, and graph-based infrastructure is more complex than a straightforward vector index.

The important lesson is not that graphs replace vectors.

It is that the retrieval structure should match the structure of the questions you need to answer.

Agentic RAG: what should the system do next?

A conventional RAG pipeline follows a predefined sequence.

An agentic RAG system gives the model more control over that sequence.

Instead of always performing exactly one retrieval call, the system can plan and decide among tools or actions. Depending on the task, those tools might include:

vector search
SQL
web search
APIs
code execution
graph traversal

A complex question might therefore become a sequence of actions:

understand the question → retrieve internal data → identify a missing piece → search another source → calculate a result → retrieve supporting evidence → synthesize the answer

This is useful when the problem is inherently multi-step or the required information lives in heterogeneous systems.

But agency introduces costs.

More tool calls mean more latency. More branches make behavior harder to predict. More moving parts make debugging harder. And an agent can make the wrong decision about what to do next.

The goal is therefore not to make every RAG system agentic.

It is to use agency when the problem actually requires adaptive,
multi-step behavior.

Four different responses to four different problems

The four patterns are easier to remember when you connect each one to the assumption it changes:

Self-RAG: Should I retrieve?
Corrective RAG: Is what I retrieved useful enough to trust?
GraphRAG: Do I need explicit relationships between pieces of knowledge?
Agentic RAG: What should I do next to solve this problem?

They can also be combined. A system might use an agent to decide whether to retrieve, use hybrid search to find candidates, apply corrective evaluation to those candidates, and use a graph when the question turns out to be relational.

The point is not to collect advanced-RAG names.

The point is to understand which limitation each architecture is trying to remove.

RAG in production is a different problem

A prototype can work beautifully on a laptop with a few documents.

Production introduces a different set of constraints.

Freshness

A knowledge base changes.

Documents are edited. Policies are replaced. New tickets arrive. Old information becomes invalid.

A production system therefore needs a strategy for freshness:

periodic re-indexing
incremental indexing triggered by document changes
timestamps and document versions
temporal ranking when recency matters
live retrieval for especially fast-changing sources

The system should know not just what a chunk says, but often when that information was valid.

Latency

A RAG request may involve query rewriting, retrieval, reranking, prompt assembly, and generation.

Every extra step adds latency.

Useful techniques include:

streaming the generated answer
caching repeated or semantically similar requests
running independent retrieval operations in parallel
using smaller models for lightweight tasks such as query rewriting
reducing the amount of context sent to the generator

Streaming is particularly important for perceived latency: users can begin reading while the model is still generating.

Cost

Every token, model call, and embedding operation has a cost.

The major levers are:

context size
model choice
number of LLM calls
cache hit rate
embedding infrastructure

A common mistake is using the most expensive model for every stage. A smaller model may be perfectly adequate for query rewriting or classification while a stronger model is reserved for the final answer.

The same principle applies to retrieval. Spending more computation on every candidate is wasteful if a fast first-stage search can narrow millions of chunks to a few dozen.

Security: the documents themselves can attack the system

Prompt injection in RAG has an unusual property.

The attacker does not necessarily need to control the user's question.

They may control a document that gets retrieved.

Imagine someone uploads a malicious support ticket containing
instructions aimed at the model. If that ticket is indexed and later retrieved, its contents are inserted into the context automatically.

The user asking the next question may be completely innocent.

This makes document trust a security boundary.

Defenses should be layered:

index content only from appropriate sources
treat user-uploaded documents as untrusted
isolate untrusted content where appropriate
preserve a clear instruction hierarchy
monitor outputs for anomalous behavior
enforce authorization before retrieval, not only after generation

Permission-aware retrieval is particularly important in multi-tenant systems.

A cache must also respect tenant and user boundaries. A perfectly cached answer is a security failure if it is returned to someone who was never authorized to see the underlying information.

Multimodal RAG

Real knowledge bases are not made entirely of paragraphs.

PDFs contain tables and charts. Wikis contain images. Technical
documentation contains diagrams.

There are several ways to handle this.

One approach is to extract or transcribe visual content into text. It is simple, but potentially lossy.

Another is to use multimodal embeddings so visual and textual content can be represented in a compatible space.

A third approach is to use a vision-language model during ingestion to generate rich descriptions of tables and images, then index those descriptions using standard text retrieval.

The practical choice depends on the application. The important point is that retrieval quality depends on preserving the information that matters, not merely converting everything into plain text.

Observability: if you can't see the pipeline, you can't debug it

A production RAG system should make its decisions inspectable.

For each query, useful telemetry can include:

the query after privacy/PII handling
retrieved chunk IDs
retrieval scores
reranking scores
prompt or prompt hash
generated answer
latency by stage
token counts
evaluation signals
user feedback

Tools such as LangSmith, TruLens, Weights & Biases, Arize AI, or a custom data warehouse can provide this visibility.

The goal is not to collect logs for their own sake.

It is to make questions like this answerable:

"Show me the queries this week where faithfulness dropped below
our baseline and tell me which retrieved chunks were involved."

That turns RAG debugging from guesswork into investigation.

RAG is not always the right answer

Understanding when not to use RAG is just as important as knowing how to build it.

If the complete knowledge base is ten pages and comfortably fits in context, retrieval may add unnecessary complexity.

If the task is purely creative, retrieval may add noise rather than useful evidence.

If the primary source is structured data, a direct database query may be better. Asking "What was Q3 revenue in APAC?" is fundamentally different from asking a question about an unstructured policy document.

And if retrieval quality is consistently poor, adding more retrieval machinery does not automatically solve the problem. Sometimes the right answer is to improve the source data or rethink the architecture.

The correct question is not:

"Should I use RAG?"

It is:

"Where does the information needed for this task live, how often does it change, and what is the best way to give the model reliable access to it?"

Designing a RAG system from requirements

When designing a RAG system, it is tempting to immediately draw a vector database and an LLM.

Start somewhere else.

Start with the requirements.

1. Understand the use case

Ask:

What kinds of documents are involved?
How large is the corpus?
How frequently does it change?
Are queries simple or multi-step?
Is the application single-turn or conversational?
What is the latency budget?
What happens if the answer is wrong?

The last question changes the architecture dramatically.

A wrong answer in a casual internal search tool is different from a wrong answer in a legal or medical workflow.

2. Design ingestion

Choose parsing and chunking based on the actual document types.

Think about deduplication, versions, deletions, metadata, permissions, and re-indexing.

3. Design retrieval

Start with a simple baseline.

Dense retrieval may be enough for some corpora.

Hybrid retrieval is often a strong general-purpose starting point when exact terms and semantic matching both matter.

Add reranking when precision matters.

Add query rewriting or decomposition when the query itself is the
bottleneck.

Do not add every technique simply because it exists.

4. Design generation

Choose the model based on the quality, latency, and cost requirements.

Define grounding behavior.

Decide how citations will work.

Define what happens when evidence is missing.

5. Design evaluation and operations

Decide how success will be measured before declaring the system
finished.

Then design for:

freshness
security
observability
caching
latency
cost
regression testing

The architecture should follow the requirements, not the other way around.

A practical example: designing RAG for a legal firm

Imagine a legal firm wants an internal assistant that answers questions about contracts, case materials, policies, and research documents.

The design immediately raises constraints.

The documents may be long and structurally complex. Access permissions matter. Citations matter. Document versions matter. An incorrect answer can have serious consequences.

That suggests several architectural choices.

Documents need careful parsing and metadata.

Chunking should preserve sections and surrounding legal context rather than blindly splitting every fixed number of tokens.

Retrieval may benefit from hybrid search because legal questions can contain both semantic concepts and exact clause numbers, names, dates, and terminology.

Reranking becomes valuable because returning the wrong clause can be worse than returning no clause.

The generation layer should be strongly grounded and citation-oriented.

Permission checks must happen before evidence reaches the model.

Evaluation should include a carefully curated golden set, not only generic synthetic questions.

And freshness/versioning cannot be an afterthought. A superseded
contract clause should not silently compete with the current version.

The interesting part of the design is not the component list.

It is the reasoning:

Each architectural decision exists because of a property of
the problem.

That is the transferable skill.

What changes when the system grows?

Scale exposes different bottlenecks.

At millions of queries, vector search throughput may become important.
At hundreds of millions of chunks, index size and memory become serious concerns.

ANN parameters can be tuned for the desired speed/recall balance. Vector compression can reduce memory usage at some accuracy cost. Large embedding pipelines need batching and parallel processing.

And cost changes character at scale.

At low traffic, an unnecessary model call is annoying.

At very high traffic, the same unnecessary model call becomes a major line item.

That is why caching, smaller models for simple tasks, efficient
retrieval, and context reduction are not merely optimization tricks.
They can determine whether the architecture is economically viable.

The deeper lesson: RAG is a system of tradeoffs

It is easy to collect RAG techniques as a list:

Chunking.

Embeddings.

Vector databases.

Hybrid search.

Reranking.

HyDE.

MMR.

GraphRAG.

Agents.

RAGAS.

And so on.

But knowing the names is not the same as understanding the system.

The more useful mental model is that every decision changes a tradeoff.

Smaller chunks can improve retrieval precision but may lose context.

Larger chunks preserve context but may dilute the relevant signal.

More retrieved documents can improve recall but increase noise and
cost.

Reranking can improve precision but adds computation.

Query rewriting can improve search quality but adds latency.

Stronger grounding can reduce unsupported claims but may increase "I don't know" responses.

Larger models can improve generation quality but increase cost and latency.

More context can provide more evidence but can also produce
attention degradation.

More sophisticated agents can solve more complex problems but are harder to predict and debug.

There is no magical configuration that wins everywhere.

The engineering question is always:

What problem are we experiencing, and which change addresses
that problem without creating a worse one somewhere else?

That is the difference between assembling a RAG demo and designing a RAG system.

RAG in one picture

At the highest level, the whole system can be reduced to a simple idea.

Documents become searchable representations.

A user's question becomes a search request.

Retrieval finds candidate evidence.

Ranking decides which evidence matters most.

The context is assembled carefully.

The LLM reasons over that evidence.

Evaluation checks whether the pipeline actually worked.

Production infrastructure keeps it fresh, fast, secure, observable, and affordable.

Final takeaway

RAG began with a simple observation:

A model does not have to contain all the knowledge it needs to answer a question. It needs a reliable way to access the right knowledge when the question arrives.

From that idea follows the entire architecture.

You need to ingest and parse the source material.

You need to chunk it without destroying meaning.

You need representations that make useful information searchable.

You need retrieval that handles both semantic similarity and exact terms.

You need ranking that separates good candidates from merely plausible ones.

You need context assembly that gives the LLM enough evidence without drowning it.

You need generation that stays grounded.

You need evaluation that tells you whether the failure happened in retrieval or generation.

And once the system becomes real, you need freshness, permissions, security, latency controls, caching, observability, and cost discipline.

The most important lesson is therefore not a particular vector database, embedding model, reranker, or RAG framework.

It is a way of thinking:

When a RAG system fails, ask where the information was lost.

Was it lost during ingestion?

During chunking?

During retrieval?

During ranking?

During context assembly?

Or during generation?

Once you can answer that question, the enormous collection of techniques around RAG becomes much easier to understand.

RAG stops looking like a bag of AI buzzwords and starts looking like what it really is:

an information pipeline designed to connect a reasoning model with the knowledge it needs.

Multithreading in Java: a guide to actually understanding it

Sarthak Rawat — Sun, 27 Sep 2026 22:56:58 +0000

The concepts that make concurrent code make sense, with enough real code that each one actually clicks.

Multithreading has a specific texture that's different from most programming topics. You can know exactly what synchronized does on paper and still get tripped up the moment you're asked to trace through what happens when two threads run the same three lines at the same time. A lot of the difficulty isn't the syntax, it's building the habit of imagining two (or ten) things happening at once, and noticing exactly where that stops being safe.

This post goes through multithreading as a concept, using Java to express it, since Java's concurrency vocabulary (synchronized, volatile, the java.util.concurrent package) is close to the vocabulary most languages end up reaching for anyway. It covers what a thread actually is, the lifecycle and the methods you'll actually use, race conditions, locks and monitors, the memory model, atomics, the executor framework, futures, concurrent collections, coordination utilities, deadlocks, virtual threads, and a way of reasoning through a stuck or misbehaving concurrent program when you're staring at one. The code targets Java 21, the current long-term support release, which matters here specifically because it's the release that made virtual threads a real, non-preview feature.

What multithreading actually is

Start simple. Multithreading means one program does more than one thing at the same time, by running several independent paths of execution instead of one. Each of those paths is a thread. Without threads, a program is a single line of instructions executing one after another; a slow file read blocks everything else, a slow network call blocks everything else, nothing overlaps. Threads let a program keep making progress on one piece of work while another piece is waiting on something slow.

Process vs thread. A process is a running program with its own private memory space, completely isolated from every other process. A thread is a path of execution inside a process, and every thread in the same process shares that process's memory. That sharing is the entire reason threads exist (they're cheap to create and can pass data to each other just by reading and writing the same variables) and it's also the entire reason multithreading is hard: two threads reading and writing the same variable at the same time is where nearly every concurrency bug traces back to.

Each thread gets its own private call stack (its own local variables and method-call history) but reads and writes the same heap as every other thread in the process. That's the picture in the cover image: shared counter, private cutting boards.

How the JVM actually runs a thread. For an ordinary Java Thread (called a "platform thread" from here on, to distinguish it from virtual threads, which work differently), the JVM maps it directly to a real operating system thread, one-to-one. The OS, not the JVM, decides when each thread actually gets CPU time and pays the cost of switching between them. That one-to-one mapping is simple and has worked for decades, but it means each thread carries real OS overhead (typically around a megabyte of stack memory, plus the cost of a context switch), which is why a server handling ten thousand concurrent connections by spinning up ten thousand platform threads runs into trouble. Virtual threads change exactly this mapping, not the language semantics around it, which is worth holding onto for now.

Creating a thread: Thread vs Runnable. Java gives you two ways to define what a thread should run.

// HelloThread.java
public class HelloThread extends Thread {
    @Override
    public void run() {
        System.out.println("Hello from " + Thread.currentThread().getName());
    }
}

// HelloTask.java
public class HelloTask implements Runnable {
    @Override
    public void run() {
        System.out.println("Hello from " + Thread.currentThread().getName());
    }
}

// Main.java
public class Main {
    public static void main(String[] args) {
        // Option 1: extending Thread directly
        HelloThread helloThread = new HelloThread();
        helloThread.start();

        // Option 2: implementing Runnable, then handing it to a Thread
        HelloTask task = new HelloTask();
        Thread t = new Thread(task);
        t.start();

        // Runnable also works fine as a lambda, since it's a single-method interface
        Thread lambdaThread = new Thread(() ->
            System.out.println("Hello from " + Thread.currentThread().getName())
        );
        lambdaThread.start();
    }
}

Runnable is the one to default to, and it's worth understanding why. Java doesn't allow multiple inheritance, so HelloThread, which extends Thread, can never extend anything else. HelloTask, which implements Runnable, stays free to extend whatever it needs to, and just as importantly, a Runnable is a plain unit of work, not tied to any particular thread, which means the exact same HelloTask object can be handed to a thread pool without any changes at all.

Thread lifecycle, states, and the methods that matter

A thread moves through a small set of states, and Java exposes them directly through Thread.getState(), which makes this an easy thing to check yourself if you're ever staring at a program that seems stuck.

NEW: the Thread object exists but start() hasn't been called yet.
RUNNABLE: the thread is either actually running or ready and waiting for the OS scheduler to give it CPU time. Java doesn't distinguish these two as separate states, both count as RUNNABLE.
BLOCKED: the thread wants a lock that another thread currently holds.
WAITING: the thread is waiting indefinitely for another thread to do something, typically via wait(), join(), or LockSupport.park().
TIMED_WAITING: same as WAITING, but with a timeout, via sleep(ms), wait(ms), or join(ms).
TERMINATED: run() has finished, normally or because of an uncaught exception.

start() vs run() is worth getting exactly right, because it's an easy thing to get quietly wrong. Calling start() asks the JVM to create a new OS thread and have it execute run() on that new thread. Calling run() directly just calls it like any ordinary method, on the current thread, synchronously, with no new thread created at all. The compiler will not stop you from writing myThread.run() instead of myThread.start(), your code will compile and even run correctly in terms of output, and it will silently not be multithreaded at all.

// StartVsRun.java
public class StartVsRun {
    public static void main(String[] args) {
        Thread t = new Thread(() ->
            System.out.println("running on: " + Thread.currentThread().getName())
        );

        t.run();    // prints "main" — ran on the calling thread, no new thread created
        t.start();  // prints "Thread-0" (or similar) — actually runs on a new thread
    }
}

The methods worth knowing beyond start(). A handful of Thread methods come up constantly once you're actually writing concurrent code, and knowing their exact behaviour, not just their names, is what makes the difference between code that looks right and code that actually is right.

join() makes the calling thread wait until the target thread finishes.

// JoinExample.java
public class JoinExample {
    public static void main(String[] args) throws InterruptedException {
        Thread worker = new Thread(() -> {
            try {
                Thread.sleep(2000); // simulate slow work
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
            System.out.println("worker finished its work");
        });

        worker.start();
        worker.join();          // main thread blocks here until worker finishes
        System.out.println("worker is done, main can continue");
    }
}

Without that join(), main has no guarantee it runs after the worker, it might print first, might print after, the order is undefined. join() also takes an optional timeout in milliseconds, after which the caller stops waiting regardless of whether the target thread finished.

sleep(ms) is a static method that pauses the current thread for at least the given duration. Two details are easy to miss here. First, it's static, so someOtherThread.sleep(1000) doesn't pause someOtherThread, it pauses whatever thread that line of code is actually running on, which trips people up more often than you'd expect. Second, and this one's genuinely important: sleep() does not release any lock the thread currently holds. A thread sleeping inside a synchronized block keeps that lock the entire time it's asleep, blocking every other thread that wants it, for the full duration.

interrupt() is Java's only real mechanism for asking a thread to stop, and it's cooperative, not forceful. Calling t.interrupt() doesn't stop the thread itself, it just sets an internal flag and, if the thread happens to be blocked in something like sleep(), wait(), or join() at that moment, wakes it up with an InterruptedException. If the thread is doing ordinary CPU work and never checks for the flag, interrupt() does nothing at all. That's why a well-behaved long-running task periodically checks whether it's been asked to stop and exits cleanly if so:

// InterruptibleTask.java
public class InterruptibleTask implements Runnable {
    @Override
    public void run() {
        while (!Thread.currentThread().isInterrupted()) {
            // do one unit of work
            performOneStep();
        }
        System.out.println("cleanly stopped");
    }

    private void performOneStep() {
        // stand-in for real work
    }
}

// Main.java
public class Main {
    public static void main(String[] args) throws InterruptedException {
        Thread t = new Thread(new InterruptibleTask());
        t.start();
        Thread.sleep(1000);   // let it run for a second
        t.interrupt();        // ask it to stop
    }
}

There is no safe way to force a thread to stop from the outside in modern Java. Thread.stop() exists but has been deprecated for a long time, because it can kill a thread mid-update and leave shared data in a half-modified, inconsistent state. Interruption is the sanctioned pattern precisely because it lets the thread finish its current unit of work and clean up before exiting, rather than being killed mid-sentence.

Daemon threads, set with t.setDaemon(true) before calling start(), are background threads the JVM won't wait for. A running JVM stays alive as long as at least one non-daemon thread is alive; the moment every remaining thread is a daemon, the JVM exits immediately, without letting those daemon threads finish whatever they were doing. Ordinary threads default to non-daemon, and this is a real, practical difference: use daemon threads for background housekeeping you're fine with being cut off abruptly, never for work that must complete. yield() is the least useful of the bunch and worth knowing mainly so you don't overestimate it: it's a hint to the scheduler that the current thread is willing to let other threads of the same priority run, but the scheduler is completely free to ignore it, and most real code never needs it.

Race conditions: the counter that doesn't count

Here's a small class with a counter, and a separate file that hammers it with ten threads, each incrementing it a thousand times:

// Counter.java
public class Counter {
    private int count = 0;

    public void increment() {
        count++;
    }

    public int getCount() {
        return count;
    }
}

// Main.java
import java.util.ArrayList;
import java.util.List;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        Counter counter = new Counter();
        List<Thread> threads = new ArrayList<>();

        for (int i = 0; i < 10; i++) {
            Thread t = new Thread(() -> {
                for (int j = 0; j < 1000; j++) {
                    counter.increment();
                }
            });
            threads.add(t);
            t.start();
        }

        for (Thread t : threads) {
            t.join();
        }

        System.out.println("Final count: " + counter.getCount());
        // Expected: 10000
        // Actual: something smaller, and different almost every run
    }
}

You would expect 10,000. You reliably get something smaller, and a different smaller number nearly every time you run it. This is the entire concept of a race condition, and the reason is that count++ isn't one operation, it's three: read the current value, add one to it, write the new value back. It's worth being able to narrate the exact interleaving that breaks it:

Thread A reads count, gets 5.
Thread A is paused by the OS scheduler before it writes anything back.
Thread B reads count, also gets 5 (A's increment hasn't been written yet).
Thread B computes 6 and writes it.
Thread A resumes, computes 6 from the value it read earlier, and writes 6 too.

Two increments happened. The counter only went up by one. Nothing crashed, nothing threw an exception, the code just silently produced the wrong answer, which is exactly what makes race conditions dangerous in real systems: they don't announce themselves. The next few sections fix this same Counter class in three different ways.

`synchronized`, monitors, and reentrancy

Java's oldest fix for the counter problem is the synchronized keyword. Every object in Java has an associated monitor (an intrinsic lock), and synchronized makes a thread acquire that lock before entering a block of code, and release it on the way out, automatically, even if an exception is thrown inside.

// Counter.java
public class Counter {
    private int count = 0;

    public synchronized void increment() {
        count++;
    }

    public synchronized int getCount() {
        return count;
    }
}

Run the exact same Main class from before, ten threads, a thousand increments each, against this version, and you get 10,000, every time. Only one thread can be inside a synchronized method (or block) on the same object at once; every other thread that wants in has to wait for the lock to be free.

You can synchronize a whole method, as above, which locks on this (the object instance), or synchronize a specific block, which lets you name exactly which object to lock on and keep the locked region as small as possible:

// Counter.java (block form instead of method form)
public class Counter {
    private int count = 0;
    private final Object lock = new Object();

    public void increment() {
        synchronized (lock) {
            count++;
        }
    }

    public int getCount() {
        synchronized (lock) {
            return count;
        }
    }
}

Smaller synchronized blocks are generally better, since anything inside the lock is a stretch of code where every other thread wanting that lock is stuck waiting, and keeping that stretch short keeps contention low.

Object-level vs class-level locking. synchronized on an instance method locks on this, so it only blocks other threads calling synchronized instance methods on that same object. Two different Counter instances don't block each other at all. If you instead want to lock across every instance of a class, for a static counter shared globally, you synchronize on the class object itself, or mark a static method synchronized, which does the same thing implicitly:

// GlobalCounter.java
public class GlobalCounter {
    private static int count = 0;

    // synchronized on GlobalCounter.class, shared by every call, from any instance
    public static synchronized void increment() {
        count++;
    }

    public static synchronized int getCount() {
        return count;
    }
}

Reentrancy is the property that lets a thread that already holds a lock re-acquire the same lock without deadlocking itself, which matters constantly in practice because one synchronized method commonly calls another synchronized method on the same object.

// ReentrantExample.java
public class ReentrantExample {
    private int count = 0;

    public synchronized void outer() {
        System.out.println("in outer(), about to call inner()");
        inner(); // the current thread already holds the lock, and re-enters safely
    }

    public synchronized void inner() {
        count++;
        System.out.println("in inner(), count is now " + count);
    }
}

If synchronized locks weren't reentrant, that call to inner() from inside outer() would deadlock the thread against itself, waiting forever for a lock it's already holding. Java's intrinsic locks are reentrant by design, and that's a deliberate design choice, not an accident, one that ReentrantLock (covered next) takes its name directly from.

`wait()`, `notify()`, and coordinating threads

synchronized stops two threads from corrupting shared data at the same time. It doesn't help when one thread needs to wait for another thread to do something first, that's a different problem, and Java's answer is wait() and notify()/notifyAll(), defined on every Object.

The rules are strict and worth memorizing exactly, because getting any one of them wrong produces a program that hangs instead of one that crashes, which is far harder to debug. wait(), notify(), and notifyAll() can only be called from inside a synchronized block on the object you're calling them on, or Java throws IllegalMonitorStateException. Calling wait() releases the lock and puts the thread to sleep until another thread calls notify() or notifyAll() on the same object. notify() wakes up one waiting thread, chosen arbitrarily; notifyAll() wakes every thread waiting on that object, and they all then compete for the lock again.

There's one rule that's very easy to get wrong: always call wait() in a loop that rechecks the condition, never in a plain if. Java allows spurious wakeups, where a thread can wake from wait() even though nobody called notify(). If your code assumes an if check before wait() is still true the moment it wakes up, you've introduced a bug that shows up rarely, unpredictably, and is miserable to reproduce.

// SafeWaitPattern.java — the shape every wait() call should follow
public class SafeWaitPattern {
    private boolean conditionIsTrue = false;
    private final Object lock = new Object();

    public void waitUntilReady() throws InterruptedException {
        synchronized (lock) {
            while (!conditionIsTrue) {   // loop, never if
                lock.wait();
            }
            // safe to proceed here, condition is genuinely true
        }
    }
}

A worked example: print odd and even alternately using two threads. This is a genuinely useful example to build once by hand, since it forces you to actually use wait()/notify() to coordinate two threads, rather than just define what they do.

// OddEvenPrinter.java
public class OddEvenPrinter {
    private int number = 1;
    private final int max;
    private boolean oddTurn = true;

    public OddEvenPrinter(int max) {
        this.max = max;
    }

    public synchronized void printOdd() {
        while (number <= max) {
            while (!oddTurn) {
                try {
                    wait();
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    return;
                }
            }
            System.out.println("Odd: " + number++);
            oddTurn = false;
            notify();
        }
    }

    public synchronized void printEven() {
        while (number <= max) {
            while (oddTurn) {
                try {
                    wait();
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    return;
                }
            }
            System.out.println("Even: " + number++);
            oddTurn = true;
            notify();
        }
    }
}

// Main.java
public class Main {
    public static void main(String[] args) {
        OddEvenPrinter printer = new OddEvenPrinter(10);

        Thread oddThread = new Thread(printer::printOdd, "odd-thread");
        Thread evenThread = new Thread(printer::printEven, "even-thread");

        oddThread.start();
        evenThread.start();
    }
}

Walk through what's actually happening: both threads call into synchronized methods on the same printer object, so only one runs at a time. The odd-thread checks a shared boolean flag; if it's not its turn, it calls wait(), releasing the lock and going to sleep. The even-thread does the same in reverse. Whichever thread does get to print flips the flag and calls notify() to wake the other one up. Neither thread ever busy-waits, burning CPU checking a condition in a loop, they genuinely sleep until told to wake up, which is the entire point of wait()/notify() over a naive spin-loop.

The Java Memory Model and `volatile`

This is one of the places where understanding what's actually happening at the hardware level pays off, because the core distinction to hold onto is that visibility and atomicity are two separate problems, and synchronized happens to solve both at once, but volatile only solves one of them.

Visibility is about whether a change one thread makes to a variable is actually seen by another thread, at all, ever. Modern CPUs have per-core caches, and a thread running on one core can update a variable in its own cache without that update being immediately, or ever, flushed to main memory where another thread on another core would see it. Without any coordination, one thread can update a boolean running = true to false, and another thread checking running in a tight loop can keep reading a stale cached true forever, spinning in a loop that should have exited.

volatile fixes exactly this, and only this. A volatile field is never cached privately; every read goes to main memory and every write goes straight back to it, so every thread always sees the latest value. It also establishes a happens-before relationship: everything a thread wrote before writing a volatile variable becomes visible to any thread that reads that same volatile variable afterward, not just the volatile field itself.

// Worker.java
public class Worker implements Runnable {
    private volatile boolean running = true;

    public void stop() {
        running = false;
    }

    @Override
    public void run() {
        int cycles = 0;
        while (running) {
            cycles++; // stand-in for real work
        }
        System.out.println("stopped after " + cycles + " cycles");
    }
}

// Main.java
public class Main {
    public static void main(String[] args) throws InterruptedException {
        Worker worker = new Worker();
        Thread t = new Thread(worker);
        t.start();

        Thread.sleep(1000);
        worker.stop(); // without volatile, this update might never be seen by t
        t.join();
    }
}

But here's the part worth being careful about: volatile gives you visibility, it does not give you atomicity. Go back to count++ and make count volatile instead of synchronizing it: the race condition is completely unchanged, because count++ is still three separate operations (read, add, write), and volatile doesn't make those three operations happen as one atomic unit. Two threads can still both read the same value before either writes, and one increment still vanishes. volatile is the right tool for a single flag one thread sets and others read, like the running boolean above. It is the wrong tool for anything involving a read-modify-write sequence, like a counter, that needs synchronized, an explicit lock, or an atomic variable instead.

Happens-before, briefly, since it's the deeper idea underneath all of this: it's the JVM's guarantee about which memory effects are visible in which order across threads. A synchronized block establishes happens-before between the thread releasing a lock and the next thread acquiring that same lock, meaning everything the first thread did before releasing is guaranteed visible to the second thread after it acquires. volatile writes and reads establish the same guarantee for that specific field. Thread.start() happens-before anything the started thread does, and everything a thread does happens-before another thread's join() on it returns. Without one of these explicit happens-before relationships in place, the JVM and the CPU are both free to reorder instructions and cache values in ways that can genuinely surprise you, which is the deeper reason unsynchronized shared-memory code is unsafe even when it "looks" correct on paper.

Explicit locks: `ReentrantLock`, `ReadWriteLock`, `StampedLock`

synchronized is easy to use correctly but inflexible. java.util.concurrent.locks gives you explicit lock objects with more control, at the cost of having to remember to release them yourself.

// Counter.java
import java.util.concurrent.locks.ReentrantLock;

public class Counter {
    private final ReentrantLock lock = new ReentrantLock();
    private int count = 0;

    public void increment() {
        lock.lock();
        try {
            count++;
        } finally {
            lock.unlock(); // must be in finally, or an exception leaves the lock held forever
        }
    }

    public int getCount() {
        lock.lock();
        try {
            return count;
        } finally {
            lock.unlock();
        }
    }
}

That finally block is not optional. With synchronized, the JVM releases the lock automatically even if an exception is thrown inside. With ReentrantLock, nothing releases it for you, so a forgotten unlock() on an exception path leaves the lock held forever, and every other thread waiting for it waits forever too.

Here's what ReentrantLock gets you that synchronized doesn't, laid out directly:

	`synchronized`	`ReentrantLock`
Acquire/release	Automatic (block-scoped)	Manual (`lock()` / `unlock()`)
Try without blocking	Not possible	`tryLock()`, returns immediately
Timeout on waiting	Not possible	`tryLock(time, unit)`
Interruptible while waiting	Not possible	`lockInterruptibly()`
Fairness (FIFO ordering)	Not possible	`new ReentrantLock(true)`
Multiple wait conditions	One implicit condition (`wait`/`notify`)	Several, via `newCondition()`

tryLock() is how you'd avoid one specific flavour of deadlock rather than just waiting forever:

// TryLockExample.java
import java.util.concurrent.locks.ReentrantLock;
import java.util.concurrent.TimeUnit;

public class TryLockExample {
    private final ReentrantLock lock = new ReentrantLock();

    public void attemptWork() {
        try {
            if (lock.tryLock(2, TimeUnit.SECONDS)) {
                try {
                    doProtectedWork();
                } finally {
                    lock.unlock();
                }
            } else {
                System.out.println("couldn't get the lock in time, doing something else instead");
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

    private void doProtectedWork() {
        // the actual work that needs the lock
    }
}

Fairness is worth a one-line caveat: a fair lock grants access in the order threads requested it, which sounds strictly better, but it comes with a real throughput cost, so it's an opt-in, not the default.

ReadWriteLock is the answer to a common, specific inefficiency: data that's read constantly and written rarely. A plain lock makes every reader wait for every other reader, even though two threads only reading the same data was never actually unsafe. ReadWriteLock splits this into a read lock, which any number of threads can hold at once as long as nobody holds the write lock, and a write lock, which is exclusive against everyone, readers included.

// SharedData.java
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;

public class SharedData {
    private final ReadWriteLock rwLock = new ReentrantReadWriteLock();
    private String data = "initial value";

    public String read() {
        rwLock.readLock().lock();
        try {
            return data;
        } finally {
            rwLock.readLock().unlock();
        }
    }

    public void write(String value) {
        rwLock.writeLock().lock();
        try {
            data = value;
        } finally {
            rwLock.writeLock().unlock();
        }
    }
}

For a read-heavy cache with occasional writes, this can meaningfully beat a plain ReentrantLock, since readers stop blocking each other entirely. StampedLock takes this further with an optimistic read mode that doesn't block writers at all, a reader just validates afterward that no write happened in the meantime and retries if one did. It's a genuinely more advanced tool, worth being able to name and describe in one sentence, but you'd typically only reach for it once you've confirmed a ReadWriteLock isn't fast enough for what you're building.

Atomic variables and compare-and-swap

There's a third way to fix the counter, and it's usually the best one for a simple case like this: skip locking entirely and use a class built for lock-free atomic updates.

// Counter.java
import java.util.concurrent.atomic.AtomicInteger;

public class Counter {
    private final AtomicInteger count = new AtomicInteger(0);

    public void increment() {
        count.incrementAndGet();
    }

    public int getCount() {
        return count.get();
    }
}

Run the same ten-threads Main class from the race condition section against this version and, again, you get exactly 10,000, with no synchronized keyword anywhere. AtomicInteger, AtomicLong, and AtomicReference work using a CPU-level instruction called compare-and-swap (CAS): "update this memory location to a new value, but only if it still holds the value I expect it to; tell me whether that succeeded."

incrementAndGet() is implemented, roughly, as a loop: read the current value, compute the new value, attempt a CAS from old to new, and if the CAS fails because another thread changed the value in between, loop around and try again with the fresh value. Here's that same loop using the lower-level compareAndSet() method directly, which shows what's happening underneath, though you'd never actually write it this way in real code, incrementAndGet() already does it for you:

// ManualCasIncrement.java — for illustration, showing what incrementAndGet() does internally
import java.util.concurrent.atomic.AtomicInteger;

public class ManualCasIncrement {
    public static void incrementManually(AtomicInteger counter) {
        while (true) {
            int current = counter.get();
            int updated = current + 1;
            if (counter.compareAndSet(current, updated)) {
                // succeeded: nothing else changed the value between get() and here
                return;
            }
            // else: another thread updated it first, loop around and retry
        }
    }
}

No thread ever blocks waiting for a lock, they just occasionally retry, which tends to perform noticeably better than locking under high contention on simple operations like this, since there's no thread ever sitting idle waiting to be woken up.

The tradeoff is scope: atomics are excellent for a single variable, a counter, a flag, a reference that gets swapped. They don't help when you need to update several related variables together as one atomic unit, like moving money between two account balances, that's still a job for a lock.

Deadlock, livelock, and starvation: a bank transfer gone wrong

A deadlock happens when two or more threads are each waiting for a lock the other one holds, and neither will ever let go. Here's the version that shows up in real code, not a toy two-lock example: two bank accounts, two threads transferring money in opposite directions at the same time.

// Account.java
public class Account {
    private final int id;
    private double balance;
    private final Object lock = new Object();

    public Account(int id, double balance) {
        this.id = id;
        this.balance = balance;
    }

    public int getId() {
        return id;
    }

    public Object getLock() {
        return lock;
    }

    public void withdraw(double amount) {
        balance -= amount;
    }

    public void deposit(double amount) {
        balance += amount;
    }
}

// BankTransfer.java — the broken version
public class BankTransfer {
    public static void transfer(Account from, Account to, double amount) {
        synchronized (from.getLock()) {
            System.out.println(Thread.currentThread().getName() + " locked account " + from.getId());
            synchronized (to.getLock()) {
                System.out.println(Thread.currentThread().getName() + " locked account " + to.getId());
                from.withdraw(amount);
                to.deposit(amount);
            }
        }
    }
}

// Main.java
public class Main {
    public static void main(String[] args) {
        Account accountA = new Account(1, 1000);
        Account accountB = new Account(2, 1000);

        Thread t1 = new Thread(() ->
            BankTransfer.transfer(accountA, accountB, 100), "Thread-1"
        );
        Thread t2 = new Thread(() ->
            BankTransfer.transfer(accountB, accountA, 50), "Thread-2"
        );

        t1.start();
        t2.start();
        // this can hang forever, depending on timing
    }
}

Walk through why this locks up: Thread 1 acquires accountA's lock first, then tries to acquire accountB's lock. At almost the same moment, Thread 2 acquires accountB's lock first, then tries to acquire accountA's lock. If the timing lines up so each thread grabs its first lock before either reaches for its second, Thread 1 is now stuck waiting for a lock Thread 2 holds, and Thread 2 is stuck waiting for a lock Thread 1 holds. Neither thread will ever release what it's holding, because releasing only happens after the synchronized block completes, and neither block can complete. The program doesn't crash, it just stops, silently, forever, which is exactly what makes deadlocks nasty in production: nothing looks wrong until it does.

The fix is lock ordering: always acquire locks in the same global order, no matter which direction the transfer is going.

// BankTransfer.java — the fixed version
public class BankTransfer {
    public static void transfer(Account from, Account to, double amount) {
        Account first = from.getId() < to.getId() ? from : to;
        Account second = from.getId() < to.getId() ? to : from;

        synchronized (first.getLock()) {
            synchronized (second.getLock()) {
                from.withdraw(amount);
                to.deposit(amount);
            }
        }
    }
}

Now every thread, regardless of which direction it's transferring, locks the lower-ID account first. Thread 1 and Thread 2 can never both be holding one lock while wanting the other, because they always reach for the same first lock, so one of them simply waits at the very first synchronized block instead of deadlocking deep inside two nested ones. The general pattern worth taking away: assign a fixed, consistent order to every lock in the system, and always acquire in that order. The other standard tool is a timeout: tryLock(time, unit) on ReentrantLock lets a thread give up and back off instead of waiting forever, which avoids the deadlock at the cost of needing retry logic.

Two related terms are worth being able to define crisply, since they're easy to confuse with deadlock and with each other. Livelock is when threads aren't blocked, they're actively running, but they keep responding to each other in a way that prevents any of them from making real progress, like two people repeatedly stepping aside for each other in a hallway and never actually passing. Starvation is when one specific thread never gets access to a resource because other threads keep getting priority over it, it's not stuck waiting on a cycle, it's just perpetually unlucky (or deprioritized). Both come up less often than deadlock, but being able to name the difference correctly reflects real understanding of the distinction rather than a memorized list of terms.

The Executor framework and thread pools

Creating a new Thread() for every unit of work doesn't scale. Each thread carries real OS overhead, and a burst of ten thousand incoming requests each spinning up its own thread will exhaust memory and grind the machine to a halt on context switching long before it exhausts CPU. The Executor framework separates submitting work from how it actually gets run, so you write task-submission code once and swap the execution strategy underneath it.

// Main.java
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Main {
    public static void main(String[] args) {
        ExecutorService executor = Executors.newFixedThreadPool(4);

        for (int i = 0; i < 10; i++) {
            int taskNumber = i;
            executor.submit(() -> {
                System.out.println("running task " + taskNumber + " on " + Thread.currentThread().getName());
            });
        }

        executor.shutdown(); // stop accepting new tasks, let submitted ones finish
    }
}

Executors offers several convenience factory methods, and it's worth knowing them, along with why most production codebases have moved away from several of them in favour of building a ThreadPoolExecutor directly:

Factory method	What it creates	The catch
`newFixedThreadPool(n)`	A pool of exactly n threads, an unbounded task queue	The unbounded queue can grow without limit if tasks arrive faster than they're processed, hiding a backlog until memory runs out
`newCachedThreadPool()`	Creates threads as needed, reuses idle ones, no upper bound	No upper bound means a burst of load can create an unbounded number of threads, exactly the problem pooling was meant to prevent
`newSingleThreadExecutor()`	One thread, tasks run strictly in order	Fine for its specific purpose, just not a general-purpose pool
`newScheduledThreadPool(n)`	Like fixed, plus delayed and repeating tasks	Same unbounded-queue concern as the fixed pool

The common thread through the catches: several of these factories quietly use an unbounded queue or an unbounded thread count, which trades one failure mode (an OutOfMemoryError from too many raw threads) for a subtler one (silent, unbounded queue growth, or the same OutOfMemoryError from an unbounded cached pool under sustained load). This is exactly why it's worth reaching for ThreadPoolExecutor directly once you actually care about these bounds, where every one of them is explicit:

// Main.java
import java.util.concurrent.*;

public class Main {
    public static void main(String[] args) {
        ExecutorService executor = new ThreadPoolExecutor(
            4,                                          // core pool size: threads kept alive even when idle
            8,                                          // maximum pool size: the ceiling under load
            60L, TimeUnit.SECONDS,                      // how long extra threads beyond core sit idle before dying
            new LinkedBlockingQueue<>(100),              // bounded queue: caps how much backlog can build up
            new ThreadPoolExecutor.CallerRunsPolicy()    // what to do when the queue is also full
        );

        for (int i = 0; i < 20; i++) {
            int taskNumber = i;
            executor.submit(() ->
                System.out.println("task " + taskNumber + " on " + Thread.currentThread().getName())
            );
        }

        executor.shutdown();
    }
}

Walk through what each parameter is actually deciding. corePoolSize threads stay alive even with nothing to do. Once the queue fills up, the pool grows beyond core, up to maximumPoolSize, to absorb the extra load. keepAliveTime controls how long those extra threads beyond core linger idle before being shut down and reclaimed. The queue itself is bounded on purpose, so backlog has a hard ceiling instead of growing invisibly. And the rejection policy decides what happens once both the pool and the queue are completely full and a new task still arrives, options include throwing an exception (AbortPolicy, the default), silently dropping the task (DiscardPolicy), or, as used above, running the task on the calling thread itself as a form of backpressure (CallerRunsPolicy), which has the effect of slowing down whoever's submitting work until the pool catches up.

Sizing a pool sensibly depends on what the work actually is. For CPU-bound work, a pool size around the number of available CPU cores is typically a reasonable starting point, since more threads than cores just means more context switching with no extra throughput. For I/O-bound work, where threads spend most of their time waiting on a network call or a disk read rather than actually using the CPU, a pool considerably larger than the core count usually performs better, since idle-waiting threads aren't competing for CPU anyway. This exact tension, threads mostly sitting idle waiting on I/O, is precisely the problem virtual threads are built to solve.

`Callable`, `Future`, and `CompletableFuture`

Runnable doesn't return anything and can't throw a checked exception. Callable<V> fixes both: it returns a value and is allowed to throw.

// Main.java
import java.util.concurrent.*;

public class Main {
    public static void main(String[] args) throws Exception {
        ExecutorService executor = Executors.newFixedThreadPool(2);

        Callable<Integer> task = () -> {
            Thread.sleep(1000);
            return 42;
        };

        Future<Integer> future = executor.submit(task);

        System.out.println("doing other work while the task runs...");
        Integer result = future.get(); // blocks until the result is ready, or throws
        System.out.println("result: " + result);

        executor.shutdown();
    }
}

future.get() blocks the calling thread until the result is available, which is useful but limited: you can't easily chain what happens next, and you can't combine the results of several futures without writing a fair amount of blocking, coordinating code by hand.

CompletableFuture solves that by letting you describe a pipeline of what should happen when a result becomes available, without ever explicitly blocking:

// Main.java
import java.util.concurrent.CompletableFuture;

public class Main {
    public static void main(String[] args) {
        CompletableFuture<Integer> future = CompletableFuture
            .supplyAsync(Main::fetchUserId)
            .thenApply(id -> id * 2)
            .thenApply(doubled -> doubled + 1);

        future.thenAccept(result -> System.out.println("Final: " + result));

        // give the async pipeline a moment to finish before the program exits
        future.join();
    }

    private static int fetchUserId() {
        // stand-in for a slow call, like hitting a database
        return 21;
    }
}

thenApply transforms a result once it's ready and hands you a new CompletableFuture wrapping the transformed value, thenAccept consumes the final result without producing a new value, and thenCompose is what you reach for when the next step is itself another asynchronous operation (chaining two CompletableFutures together rather than nesting one inside a plain function). CompletableFuture.allOf(f1, f2, f3) waits for several independent futures to all complete, which is the standard way to fan out several concurrent calls, say, three separate API calls, and combine their results once every one of them is done.

Concurrent collections, and building a blocking queue by hand

Wrapping a plain HashMap or ArrayList in synchronized on every call works, but it serializes every single access, one thread at a time, even for operations that could safely happen concurrently. java.util.concurrent gives you collections designed for concurrent access from the ground up, and they're almost always the better choice.

ConcurrentHashMap allows concurrent reads and writes without locking the entire map for every operation, internally splitting its locking so unrelated keys don't contend with each other. It's the default choice for a shared map accessed by multiple threads, essentially never Collections.synchronizedMap(new HashMap<>()) in new code.

// Main.java
import java.util.concurrent.ConcurrentHashMap;
import java.util.Map;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        Map<String, Integer> visitCounts = new ConcurrentHashMap<>();

        Runnable visitor = () -> {
            for (int i = 0; i < 1000; i++) {
                visitCounts.merge("home-page", 1, Integer::sum);
            }
        };

        Thread t1 = new Thread(visitor);
        Thread t2 = new Thread(visitor);
        t1.start();
        t2.start();
        t1.join();
        t2.join();

        System.out.println(visitCounts.get("home-page")); // reliably 2000
    }
}

CopyOnWriteArrayList takes a different approach entirely: every write creates a brand-new copy of the underlying array, while reads always work against a stable, never-changing snapshot and need no locking at all. That makes it excellent for read-heavy, write-rare situations, like a list of event listeners that's iterated constantly and modified rarely, and a poor choice for anything write-heavy, since copying the entire array on every single write gets expensive fast.

BlockingQueue implementations (ArrayBlockingQueue, LinkedBlockingQueue) add blocking behaviour on top of a queue: put() blocks the calling thread if the queue is full instead of failing, and take() blocks if the queue is empty instead of returning null. That blocking behaviour is exactly what makes the producer-consumer pattern simple to write correctly instead of needing manual wait()/notify() coordination.

It's worth actually building a small bounded blocking queue yourself once, using nothing but wait()/notify(), because it's a great way to see exactly what BlockingQueue is doing underneath rather than just knowing it exists:

// SimpleBlockingQueue.java
import java.util.LinkedList;
import java.util.Queue;

public class SimpleBlockingQueue<T> {
    private final Queue<T> queue = new LinkedList<>();
    private final int capacity;

    public SimpleBlockingQueue(int capacity) {
        this.capacity = capacity;
    }

    public synchronized void put(T item) throws InterruptedException {
        while (queue.size() == capacity) {
            wait(); // full, wait for a consumer to make room
        }
        queue.add(item);
        notifyAll(); // wake any consumer waiting on take()
    }

    public synchronized T take() throws InterruptedException {
        while (queue.isEmpty()) {
            wait(); // empty, wait for a producer to add something
        }
        T item = queue.poll();
        notifyAll(); // wake any producer waiting on put()
        return item;
    }
}

// Main.java
public class Main {
    public static void main(String[] args) {
        SimpleBlockingQueue<Integer> queue = new SimpleBlockingQueue<>(5);

        Thread producer = new Thread(() -> {
            try {
                for (int i = 0; i < 20; i++) {
                    queue.put(i);
                    System.out.println("produced " + i);
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });

        Thread consumer = new Thread(() -> {
            try {
                for (int i = 0; i < 20; i++) {
                    int item = queue.take();
                    System.out.println("consumed " + item);
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });

        producer.start();
        consumer.start();
    }
}

Both put() and take() wait in a loop, not an if, for exactly the spurious-wakeup reason covered earlier. notifyAll() rather than notify() matters here specifically because both producers and consumers might be waiting on the same object at once, notify() would only wake one arbitrary thread, which might not even be the kind of thread that can currently make progress; notifyAll() wakes everyone and lets each one recheck its own condition. This is, in miniature, exactly what ArrayBlockingQueue does internally, just with more tuning and edge-case handling than is worth reproducing by hand in real code.

Coordination utilities: `CountDownLatch`, `CyclicBarrier`, `Semaphore`

These three exist for a specific, recurring shape of problem: several threads need to coordinate their timing with each other, not just their access to shared data.

CountDownLatch lets one or more threads wait until a fixed number of events have happened. It's initialized with a count, other threads call countDown() as they finish their part, and any thread that called await() unblocks the moment the count hits zero. It cannot be reset or reused once it reaches zero, it's a strictly one-time gate.

// Main.java
import java.util.concurrent.CountDownLatch;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        CountDownLatch latch = new CountDownLatch(3);

        for (int i = 0; i < 3; i++) {
            int workerId = i;
            new Thread(() -> {
                System.out.println("worker " + workerId + " starting work");
                doWork();
                System.out.println("worker " + workerId + " finished");
                latch.countDown();
            }).start();
        }

        latch.await(); // blocks until all 3 threads have called countDown()
        System.out.println("all workers finished, main can continue");
    }

    private static void doWork() {
        try {
            Thread.sleep(500);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

CyclicBarrier looks similar but solves a different shape of problem: instead of one thread waiting for several others, a group of threads all wait for each other to reach the same point before any of them proceeds, and once released, the barrier resets and can be used again for the next round.

// Main.java
import java.util.concurrent.CyclicBarrier;

public class Main {
    public static void main(String[] args) {
        CyclicBarrier barrier = new CyclicBarrier(3,
            () -> System.out.println("all 3 threads reached the barrier, proceeding together")
        );

        for (int i = 0; i < 3; i++) {
            int workerId = i;
            new Thread(() -> {
                doWorkPhaseOne(workerId);
                try {
                    barrier.await();
                } catch (Exception e) {
                    Thread.currentThread().interrupt();
                }
                doWorkPhaseTwo(workerId); // only starts once all 3 threads reached the barrier
            }).start();
        }
    }

    private static void doWorkPhaseOne(int workerId) {
        System.out.println("worker " + workerId + " doing phase one");
    }

    private static void doWorkPhaseTwo(int workerId) {
        System.out.println("worker " + workerId + " doing phase two");
    }
}

The concrete difference worth taking away: CountDownLatch is for one thread (or several) waiting on a fixed number of other threads to finish, used once; CyclicBarrier is for a fixed group of threads waiting on each other to all reach the same checkpoint together, and it's reusable round after round, useful for something like a simulation that proceeds in synchronized phases.

Semaphore controls access to a resource with a limited number of permits, rather than the single all-or-nothing lock that synchronized and ReentrantLock give you. acquire() takes a permit, blocking if none are available; release() gives one back. A semaphore initialized with 3 permits allows exactly three threads through at once, a straightforward way to cap concurrent access to something like a pool of database connections or a rate-limited external API.

// Main.java
import java.util.concurrent.Semaphore;

public class Main {
    public static void main(String[] args) {
        Semaphore semaphore = new Semaphore(3); // at most 3 concurrent

        for (int i = 0; i < 10; i++) {
            int callId = i;
            new Thread(() -> {
                try {
                    semaphore.acquire();
                    System.out.println("call " + callId + " got a permit");
                    callLimitedResource();
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                } finally {
                    semaphore.release();
                }
            }).start();
        }
    }

    private static void callLimitedResource() {
        try {
            Thread.sleep(500);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

Virtual threads: what changed in Java 21

This is worth understanding properly, because virtual threads became a standard, non-preview feature in Java 21 (they'd been previewed in 19 and 20 first), and they've genuinely changed how high-throughput Java servers get built.

A traditional web server handles each incoming request on its own platform thread, and each platform thread is a real OS thread, carrying real memory overhead and real scheduling cost. Most of a typical request's time isn't spent computing, it's spent waiting: on a database query, on a downstream API call, on disk I/O. A platform thread sits there fully allocated, doing nothing, for the entire wait. Scale that to tens of thousands of concurrent requests and you run out of threads, and the machine, long before you run out of actual CPU work to do.

A virtual thread is a thread implemented by the JVM itself rather than mapped one-to-one to an OS thread. Many thousands of virtual threads run on top of a small pool of ordinary platform threads, called carrier threads. When a virtual thread blocks on I/O, the JVM unmounts it from its carrier thread entirely, freeing that carrier to run a different virtual thread, and remounts the original one onto some carrier once its I/O actually completes. The blocking, from the code's point of view, looks exactly like ordinary blocking code always has, Thread.sleep(), a blocking database call, a synchronous HTTP client, none of it needs to be rewritten in a reactive or callback style. That's the actual headline: you get the throughput characteristics of async code while writing plain, ordinary, sequential-looking blocking code.

// Main.java
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Main {
    public static void main(String[] args) {
        try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) {
            for (int i = 0; i < 10_000; i++) {
                int taskId = i;
                executor.submit(() -> {
                    makeSlowNetworkCall(); // blocks this virtual thread, not a whole OS thread
                    return null;
                });
            }
        } // executor.close() is called automatically here, waiting for all tasks to finish
    }

    private static void makeSlowNetworkCall() {
        try {
            Thread.sleep(100); // stand-in for a real I/O-bound call
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

Ten thousand platform threads would very plausibly crash a typical server. Ten thousand virtual threads, each cheap to create and cheap to sit idle, generally won't, because the JVM never had to allocate ten thousand OS threads' worth of stack memory and scheduling overhead to begin with.

The one caveat worth holding onto clearly: virtual threads help I/O-bound workloads, where threads spend most of their time waiting. They do essentially nothing for CPU-bound work, tight loops doing real computation with no blocking at all, because in that case there's no idle waiting time to reclaim in the first place; the bottleneck is genuinely the CPU, and virtual threads don't create more CPU cores. It's also worth knowing that two closely related Loom features, structured concurrency and scoped values, are still preview features even as of Java 25, not yet stable APIs, so it's fine to know what they're for in one sentence each (structured concurrency treats a group of related subtasks as a single unit that succeeds or fails together; scoped values are a safer, immutable alternative to ThreadLocal for sharing context across threads) without needing to write their exact preview syntax from memory.

Producer-consumer, done properly

Here's producer-consumer done once, cleanly, with the real BlockingQueue rather than the hand-rolled SimpleBlockingQueue from before.

// Main.java
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;

public class Main {
    public static void main(String[] args) {
        BlockingQueue<Integer> queue = new ArrayBlockingQueue<>(10);

        Thread producer = new Thread(() -> {
            try {
                for (int i = 0; i < 100; i++) {
                    queue.put(i); // blocks automatically if the queue is full
                    System.out.println("produced " + i);
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });

        Thread consumer = new Thread(() -> {
            try {
                while (true) {
                    Integer item = queue.take(); // blocks automatically if the queue is empty
                    System.out.println("consumed " + item);
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });

        producer.start();
        consumer.start();
    }
}

Notice how much of the earlier complexity has simply disappeared. No synchronized, no manual wait()/notify(), no explicit condition-checking loop, BlockingQueue has absorbed all of it internally, using essentially the same wait-loop-and-notify mechanics behind SimpleBlockingQueue. That's the real lesson here: the primitives (synchronized, wait, notify) are what you need to understand deeply, so you can reason about what's happening underneath and debug it when it breaks, but the higher-level tools (BlockingQueue, ExecutorService, CompletableFuture) are almost always what you should actually reach for in real code, since they've already handled the edge cases you'd otherwise get wrong.

Reasoning through a stuck or misbehaving program

There's one more skill worth building: being able to look at a concurrent program that's misbehaving and reason through why, methodically, rather than guessing. Here's a way to think about it that covers most real situations.

A program appears to hang. The first, fastest diagnostic is a thread dump, triggerable with jstack <pid> from the command line, or by sending SIGQUIT to the process, or by capturing one straight from your IDE's debugger. A thread dump lists every thread in the JVM along with its current state and, critically, its full stack trace at that exact instant. Scan specifically for threads in the BLOCKED state, that tells you they're waiting on a lock, and the dump names exactly which lock and which thread is currently holding it. If thread A is BLOCKED waiting for a lock thread B holds, and thread B's stack shows it BLOCKED waiting for a lock thread A holds, you've found your deadlock directly in the dump, no guessing required. Modern JVMs actually detect this specific cycle for you and print "Found one Java-level deadlock" right at the top of the dump, naming the exact threads and locks involved.

A program produces wrong results, but doesn't hang. This is the race-condition case, and it's harder precisely because nothing crashes and nothing blocks; the program simply computes something subtly incorrect, and often only under real concurrent load, never in a quick single-threaded test. The systematic approach is to look for exactly the pattern from the counter example: any place where a shared, mutable variable is read, modified, and written back without a lock, an atomic type, or a volatile covering the whole read-modify-write sequence. count++, list.add() on a plain ArrayList, if (map.get(k) == null) map.put(k, v), that check-then-act pattern in particular is a very common, very real source of race conditions, since the check and the act aren't atomic together even if each individually looks safe.

A program is slow, but not stuck. Here a thread dump is still the first move, but you're looking for something different: threads sitting RUNNABLE for a long stretch doing real work, one very large synchronized block that most other threads are BLOCKED waiting on, or, in older code, contention around a single coarse lock that could be split into a ReadWriteLock, a ConcurrentHashMap, or several finer-grained locks instead of one lock guarding far more than it needs to.

The consistent thread through all three: pin down the symptom precisely first, hung completely, wrong output, or just slow, because each symptom points toward a genuinely different class of cause, and starting from that distinction is almost always faster than diving straight into a specific fix and hoping.

Docker for SDE interviews: the guide I wish I'd had

Sarthak Rawat — Fri, 25 Sep 2026 22:57:26 +0000

What it actually does under the hood, the Dockerfile traps interviewers love, and the debugging playbook that makes you sound like you've shipped something.

Docker questions show up in almost every backend interview now, even ones that aren't about infrastructure. You mention deploying something, and the interviewer asks how you packaged it. You draw a system design diagram, and there's a box labeled "container" that nobody explains. Half the time it's a genuine technical question. The other half it's a filter: does this person actually understand what they've been typing into a terminal, or have they only ever copied a Dockerfile from Stack Overflow?

This post covers what you actually need: how Docker works under the hood, writing a Dockerfile that isn't naive, images and layers, networking, storage, Compose, the CLI commands worth knowing, security basics, where Docker stops being enough, and a debugging playbook for the scenario questions interviewers like to throw at you.

What Docker actually is

Here's the confusion to clear up first, because it trips up more candidates than anything else: a container is not a lightweight virtual machine. It's a regular process on the host's Linux kernel, made to look isolated using three kernel features that have existed for years, which Docker packaged into something usable.

Namespaces give a process its own view of things that are normally global. A PID namespace makes a container's first process think it's PID 1, even though the host sees it as PID 48213. A network namespace gives it its own network interfaces and routing table. There are namespaces for mount points, hostnames, user IDs, and inter-process communication too. Namespaces are about what a process can see.

Control groups (cgroups) limit what a process can use: how much CPU, memory, and I/O bandwidth. docker run --memory=512m is a cgroup limit. Without one, a single container can eat all the host's RAM and take everything else down with it, which is exactly the kind of thing that happens in production and gets asked about in interviews.

A union filesystem stacks read-only image layers with one thin writable layer on top, so containers share the same base files on disk instead of each getting a full copy.

Put those three together and you get something that starts in milliseconds, because you're not booting a kernel, just starting a process with some walls around it.

That's also the honest answer to "container vs VM," which is asked in nearly every Docker interview. A VM virtualizes hardware: a hypervisor gives each VM its own kernel, so a VM can run a completely different OS than its host, and boots in the order of a minute because it's booting an entire operating system. A container virtualizes at the OS level: it shares the host's kernel, so a Linux container needs a Linux host kernel underneath it (Docker Desktop on Mac and Windows quietly runs a small Linux VM to give you that kernel), and it starts in milliseconds to a couple of seconds because there's no kernel boot involved. The tradeoff is isolation strength: a VM's hypervisor boundary is harder to break out of than a container's kernel-feature boundary, which is part of why nobody runs genuinely hostile, untrusted code in a bare container without extra layers like gVisor or Kata Containers on top.

This is also why "Docker Desktop" and "Docker Engine" aren't quite the same thing, a distinction worth having straight if you develop on a Mac or Windows laptop and deploy to Linux servers, which describes most people. On Linux, Docker Engine runs natively, talking directly to the host kernel's namespaces and cgroups. On Mac and Windows there is no Linux kernel to talk to, so Docker Desktop quietly runs a small Linux VM in the background and Docker Engine runs inside that VM instead. Functionally you barely notice the difference day to day, but it's the reason volumes on a Mac live inside that hidden VM rather than as a directly browsable folder on your actual filesystem, and it's part of why "works fine on my Mac, does something weird on the server" occasionally isn't actually a lie, there's a real extra layer on your laptop that the server doesn't have.

The architecture. Four pieces, and interviewers like to hear you name all four because it explains why Docker commands sometimes fail in confusing ways. The Docker CLI is the docker command you type. The Docker daemon (dockerd) is a background service that does the actual work: building images, running containers, managing networks and volumes. The Docker Engine is the daemon plus its REST API and the CLI together, the whole toolset. A registry stores and distributes images; Docker Hub is the default public one. When you run docker build, the CLI doesn't build anything itself, it sends a request to the daemon over that REST API, and the daemon does the work. That's why "Docker isn't working" is so often really "the daemon isn't running" or "my user doesn't have permission to reach it," and knowing that split is what separates someone who's memorized commands from someone who understands the tool.

One more distinction worth locking in early, because interviewers ask it directly: an image is a static, read-only blueprint, built once and then reused. A container is a running instance of that image, a live process with its own writable layer on top. One image can spin up many containers at once, each isolated from the others, the way one class definition can produce many objects. A volume is neither of those. It's a mechanism for storing data outside a container's writable layer, so that data survives when the container is removed. We'll come back to volumes properly in the storage section, but keep the three separate in your head: image is the recipe, container is the meal, volume is the fridge you keep leftovers in after the meal's plate gets thrown away.

Images, layers, and registries

An image is a stack of read-only layers plus some metadata (which command starts the container, which ports it documents, what its default working directory is). Each instruction in a Dockerfile that changes the filesystem produces one layer, and each layer is content-addressed: Docker hashes its contents, and if two images share a layer with the same hash, they share the actual bytes on disk instead of duplicating them. That's why pulling a new image that shares a base with one you already have is often fast, you're only downloading the layers you don't already have.

This is the whole logic behind the build cache, and it's the single most common "do you actually understand Docker" test. Docker builds a Dockerfile top to bottom, and before running each instruction it checks whether it's seen that exact instruction with the exact same inputs before. If so, it reuses the cached layer instead of redoing the work. The moment one instruction changes, every instruction after it has to rerun, even if nothing else changed, because each layer builds on top of the one before it. That's why you copy requirements.txt and install dependencies before copying the rest of your application code: your code changes on every commit, but your dependencies don't, so if you copy code first, every build reinstalls every dependency from scratch. Order layers from least-likely-to-change to most-likely-to-change, and your builds stay fast for months.

# Slow: any code change reinstalls every dependency
COPY . .
RUN pip install -r requirements.txt

# Fast: dependency layer stays cached until requirements.txt itself changes
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

Storage and registries. Images live locally under Docker's data directory until you push them somewhere. A registry is just a storage and distribution service for images: docker pull fetches from one, docker push sends to one. Docker Hub is the default and where most official base images live (python, node, postgres, nginx). In a real company you'll almost always be pushing to a private registry instead, like Amazon ECR, Google Artifact Registry, GitHub Container Registry, or a self-hosted one like Harbor, so images never leave your own infrastructure.

Tagging is where people get careless, and it's a favourite interview trap. An image reference is name:tag, and if you don't specify a tag, Docker assumes latest. The trap is that latest doesn't mean "the newest version," it's just a tag like any other that happens to be the default. If you keep pushing new builds tagged latest, the previous image becomes an untagged, unreferenced layer sitting on disk somewhere, and you've lost your ability to roll back to it by name. In production you want an explicit version or, more commonly now, the git commit SHA as the tag: myapp:a3f9c21. That gives you full traceability. You always know exactly which commit is running in any environment, and rolling back means simply redeploying the previous SHA.

Writing a Dockerfile that isn't naive

A Dockerfile is a plain text file of instructions Docker follows, top to bottom, to build an image. Here's a reasonably real one for a small Python service:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["python", "app.py"]

Walking through what each line is actually doing: FROM sets the base image everything else builds on top of, here a slim, minimal Python image rather than the full one, which matters for size (more on that shortly). WORKDIR sets the working directory inside the image for every instruction after it, and creates the directory if it doesn't exist. COPY copies files from your build context into the image; RUN executes a command at build time and bakes its result into a new layer. EXPOSE is worth being precise about, because it's a common source of confusion in interviews: it does not publish a port. It's purely documentation, a note in the image's metadata saying "this app listens on 8000." The thing that actually makes a port reachable from outside the container is the -p flag at runtime, docker run -p 8080:8000 myapp, which maps host port 8080 to container port 8000. You can EXPOSE a port and never publish it, and you can publish a port you never bothered to EXPOSE. CMD sets the default command the container runs when it starts.

COPY vs ADD. Both copy files into the image, but ADD quietly does more: it auto-extracts local .tar archives into the destination, and it can fetch files from a remote URL. That second behaviour is exactly why most teams avoid it: pulling from a URL at build time means your build isn't reproducible, since the same Dockerfile can produce a different image tomorrow if that URL's contents change. Use COPY by default. Reach for ADD only on the rare occasion you specifically need local tar extraction.

ENV vs ARG. Both set variables, but their lifetimes differ. ARG exists only during the build and isn't present in the final image or in a running container, useful for things like choosing a base image version at build time. ENV sets an environment variable that persists into the running container, so anything your application reads from the environment at runtime should be ENV, not ARG. A subtlety worth knowing: ARG values do show up in docker history, so never pass secrets through build args either, they're not hidden, just short-lived.

CMD vs ENTRYPOINT is the classic trap, and it's worth getting exactly right. Both define what runs when a container starts, but they behave differently when you pass arguments at runtime. CMD sets a default command that gets fully replaced if you specify anything after the image name in docker run. ENTRYPOINT sets a fixed executable that always runs; anything you pass at runtime gets appended to it as arguments instead of replacing it.

# CMD alone — fully overridable
CMD ["python", "app.py"]
# docker run myimage python other.py  →  runs "python other.py", ignoring app.py entirely

# ENTRYPOINT + CMD — entrypoint is fixed, CMD is just its default argument
ENTRYPOINT ["python"]
CMD ["app.py"]
# docker run myimage other.py  →  runs "python other.py"
# docker run myimage           →  runs "python app.py"

The common, sensible pattern is ENTRYPOINT for the fixed executable and CMD for a default argument you're happy to have overridden. If you genuinely need to override the entrypoint itself at runtime, there's an explicit --entrypoint flag for that; it doesn't happen by accident. For a lot of simple apps CMD alone is all you need, and ENTRYPOINT only earns its place when you want to lock in the executable and treat everything else as swappable arguments.

There's a second, quieter trap sitting right next to this one: the JSON array form above, CMD ["python", "app.py"], is called exec form, and it matters for more than syntax. Write it instead as a plain string, CMD python app.py, and Docker treats it as shell form, silently wrapping it as /bin/sh -c "python app.py". That means your app isn't actually PID 1 inside the container, the shell is, with your app running as a child process underneath it. When docker stop sends SIGTERM, it goes to PID 1, the shell, which generally has no idea it's supposed to forward that signal on to your app. Your app never hears about the shutdown, so it just keeps running until the grace period expires and Docker escalates to SIGKILL, meaning every stop takes the full timeout and nothing gets a clean chance to exit. Exec form avoids all of this because your app becomes PID 1 directly and receives signals itself. Default to exec form for both CMD and ENTRYPOINT unless you have a specific reason to want the shell in between, like needing shell features such as environment variable expansion in the command itself.

The build context and .dockerignore. When you run docker build ., that trailing . is the build context: the entire directory Docker sends to the daemon before the build even starts. Every file in it gets transferred, whether your Dockerfile uses it or not, which matters for two reasons. A large context slows down every single build, since it all has to ship over before anything happens. And only files inside the context are available to COPY; anything outside it is invisible to the build. A .dockerignore file works exactly like .gitignore and trims the context down, typically excluding .git, virtual environments, local caches, test directories, and any build artifacts.

Multi-stage builds and cutting image size

Multi-stage builds are the answer to a genuinely common interview scenario: "here's a 1.2 GB image, get it under 200 MB." A Dockerfile can have more than one FROM, and each one starts a fresh stage. You can copy specific files from an earlier stage into a later one, and the final image only contains whatever the last stage explicitly copied in. Compilers, build tools, and test dependencies from earlier stages never make it into the image that actually ships.

# Stage 1: build, with all the compiler tooling you need
FROM python:3.12 AS builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --target=/build/packages -r requirements.txt

# Stage 2: the image that actually ships
FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /build/packages /usr/local/lib/python3.12/site-packages
COPY . .
CMD ["python", "app.py"]

The savings are most dramatic for compiled languages, where a Go or Java build stage can drag in gigabytes of toolchain that a runtime image never touches. For an interpreted language like Python the main win is excluding build-time-only packages like gcc, needed to compile certain dependencies but useless afterward.

The other lever, alongside multi-stage builds, is your base image choice. python:3.12 is roughly 900 MB. python:3.12-slim strips out most tools you don't need at runtime and lands around 130 MB. Alpine-based images go smaller still, but Alpine uses musl instead of glibc, which occasionally breaks Python packages with compiled C extensions, so it's not a free upgrade; slim is usually the safer default. Beyond slim there are distroless images, which strip out even the shell and package manager, leaving close to nothing but your app and its runtime, the smallest attack surface but also the hardest to debug, since you can't exec into a shell that isn't there. Combine a slim or distroless final stage with multi-stage builds and a .dockerignore, and a 1.2 GB image getting down to under 200 MB is a completely normal outcome, not a special trick.

Container lifecycle and the essential CLI

Before the states themselves, one thing worth being clear on, because it's the source of a very common fresher confusion: docker run ubuntu with nothing else after it starts a container and it exits immediately, often before you've even finished reading the output. This looks broken, but it isn't. A container stays alive for exactly as long as its main process, PID 1, stays alive, and nothing else. The ubuntu image's default command is a shell with no terminal attached to keep it open, so that shell starts, has nothing to do, and exits, and the moment PID 1 exits, the whole container stops, regardless of anything else that happened to be running. This is genuinely different from a VM, where the "machine" stays up as its own thing independent of whatever process you happen to be running inside it. A container has no concept of "staying up" separate from its main process; the process is the container's lifetime.

This is also why you'll rarely see a well-designed container running more than one real service. It's tempting, especially early on, to think "why not run nginx and my app and a cron job all in one container," but Docker only supervises one PID 1, so if you cram three services into one entrypoint script, Docker has no idea if two of them silently died, it only knows whether the wrapper script itself is still alive. The convention, and the thing to say in an interview if it comes up, is one process per container: if you need several services, run several containers and let Compose or an orchestrator manage them as a group, each with its own lifecycle Docker can actually see and act on.

With that settled, a container moves through a small number of states, and knowing the exact commands and signals for each transition is table stakes.

State	How you get there	What's happening
Created	`docker create`	Filesystem is set up from the image, nothing is running yet
Running	`docker run` (create + start) or `docker start`	The main process is executing
Paused	`docker pause`	Processes are frozen in place, memory held
Stopped	`docker stop` or `docker kill`	Main process has exited, filesystem still on disk
Removed	`docker rm`	Container and its writable layer are gone

docker run and docker start get mixed up constantly, so it's worth being precise: docker run always creates a brand-new container from an image and starts it. docker start restarts a container that already exists but is currently stopped. If you docker run the same image five times, you get five separate containers; if you docker start a stopped one, you get the same container back, with the same writable layer and any data it had.

docker stop and docker kill also aren't the same thing, and interviewers like this pairing because it has a real production consequence. docker stop sends SIGTERM, gives the process a grace period (10 seconds by default) to shut down cleanly, and only sends SIGKILL if it hasn't exited by then. docker kill sends SIGKILL immediately, no grace period, no chance to flush a buffer or finish a write. For anything talking to a database mid-transaction, that difference matters.

Containers don't restart on their own unless you tell them to, via --restart:

Policy	Behaviour
`no`	Never restart automatically (the default)
`always`	Restart whenever it stops, even after the Docker daemon itself restarts
`on-failure[:N]`	Restart only on a non-zero exit code, optionally capped at N retries
`unless-stopped`	Like `always`, but won't restart a container you stopped manually

unless-stopped is usually the sensible production default, since always will happily restart a container you stopped for maintenance the moment the daemon comes back up, which is rarely what you want.

Exit codes are worth glancing at when a container has already died: 0 is a clean exit, and 137 specifically means the process was killed by SIGKILL, most often because the kernel's OOM killer stepped in after the container hit its memory limit. Seeing 137 in docker ps -a should make you reach straight for docker stats and your memory limits, not the application logs.

Here's the CLI reference worth actually knowing, not memorizing flag-by-flag, but knowing what each one is for and why you'd reach for it:

Command	What it's for
`docker build -t name .`	Build an image from a Dockerfile, tagging it `name`
`docker run -d -p 8080:80 name`	Create and start a container, `-d` runs it detached in the background instead of tying up your terminal in the foreground
`docker run --env-file .env name`	Load many environment variables from a file at once, instead of a long chain of `-e KEY=value` flags
`docker ps` / `docker ps -a`	List running containers / all containers including stopped ones
`docker exec -it name sh`	Run a new process (usually a shell) inside a running container, for debugging
`docker logs --tail 50 -f name`	Show recent stdout/stderr, follow live with `-f`
`docker cp file.txt name:/app/`	Copy a file between the host and a running container, in either direction, without needing a shell
`docker stop` / `docker kill`	Graceful shutdown (SIGTERM) vs immediate (SIGKILL)
`docker rm` / `docker rmi`	Remove a container / remove an image
`docker inspect name`	Full JSON metadata: IPs, mounts, env vars, health status, everything
`docker stats`	Live CPU, memory, and I/O usage per container
`docker pull` / `docker push`	Fetch an image from a registry / send one to a registry
`docker image prune` / `docker system prune`	Clean up dangling images / clean up all unused containers, networks, and images

A couple of these are worth a sentence more than the table gives them. -d matters because without it your terminal is attached to the container's output and blocked until it exits, fine for a quick test, useless for anything you want running in the background while you keep working. --env-file is the practical way to pass a real application's worth of configuration, database URLs, API keys, feature flags, without a docker run command that's twenty -e flags long; Compose has the equivalent env_file: key for the same reason. docker cp is the one people forget exists and then do something roundabout instead, like rebuilding an image just to add one file, when copying it straight into a running container takes one command.

One easy-to-miss distinction: docker exec starts a brand-new process inside an already-running container, which is what you want for debugging, poking around, or running a one-off script. docker attach instead connects your terminal directly to the container's main process (PID 1). If you Ctrl+C out of an attached session instead of detaching properly with Ctrl+P, Ctrl+Q, you can send a kill signal straight to that main process and stop the container by accident. For everyday debugging, exec is almost always the right tool.

A last cleanup note that catches people running CI pipelines: a dangling image is an old, untagged image layer left behind when you rebuild an image with the same name and tag, orphaning the previous version. They pile up quietly and can fill a CI runner's disk within days if nothing ever cleans them up. docker image prune handles just the dangling ones; docker system prune -a is far more aggressive and removes every image not currently used by a running container, so use it carefully.

Networking

By default, Docker gives you a bridge network, a private virtual network on the host that containers attach to. The default bridge is fairly limited: containers on it can only reach each other by IP address. A user-defined bridge network, which you create yourself with docker network create, is much more useful, because Docker runs an embedded DNS server on it, so containers can resolve each other by name instead of by IP. This is exactly what makes Docker Compose feel effortless: every service in a Compose file gets attached to the same user-defined network automatically, and a web service can just connect to postgres://db:5432 using db as a hostname, no IP addresses anywhere.

There are four network drivers worth knowing:

Bridge (the default, described above): single-host, containers reach each other by name on a shared virtual network.
Host: the container skips network isolation entirely and shares the host's network stack directly. Fastest possible networking, since there's no virtual bridge in between, but no port mapping and no isolation, since the container binds directly to host ports.
None: no networking at all, useful for fully isolated batch jobs that need zero network access.
Overlay: extends a virtual network across multiple Docker hosts, which is what makes Swarm's multi-host clustering possible in the first place. You won't reach for this on a single machine.

The one detail that trips people up in a real debugging scenario: inside a container, localhost refers to that container itself, not to another container or to the host machine. If service A needs to reach service B, it has to use B's container name (or Compose service name) on a shared user-defined network, localhost will just fail silently or connect to nothing.

Port publishing is worth restating cleanly here since it connects back to EXPOSE: -p 8080:8000 maps host port 8080 to container port 8000, and that's the only thing that actually makes a container reachable from outside Docker's network. Nothing else does it.

Storage: volumes, bind mounts, and tmpfs

Anything written to a container's writable layer disappears the moment that container is removed. That's fine for a stateless web server, and a real problem for a database. Docker gives you three ways to persist or share data, and interviewers care about you knowing which one fits which situation.

Type	Where it lives	Survives container removal?	Best for
Named volume	Docker-managed location on the host	Yes	Databases, production data
Bind mount	A specific path you choose on the host	Yes, it's just the host filesystem	Local development, live code reload
tmpfs	RAM only, never touches disk	No, gone the moment the container stops	Secrets, sensitive temp data

A named volume (docker volume create my_data, then docker run -v my_data:/app/data) is storage Docker manages for you, outside of any specific container's filesystem, kept at /var/lib/docker/volumes/ on Linux. It's the right default for a database or anything you'd genuinely be upset to lose, and it's independent of any one container's lifecycle. On macOS and Windows, Docker Desktop runs the engine inside a lightweight Linux VM under the hood, so volumes live inside that VM rather than as a directly browsable host path, worth knowing so you're not confused when you go looking for the files and they're not where you expected.

A bind mount links a specific path on the host directly into the container: docker run -v $(pwd):/app. It's what makes local development pleasant, since editing a file on your host shows up inside the running container instantly, no rebuild needed. The tradeoff is that it depends entirely on your host's directory layout and gives the container direct read/write access to a real host path, which is a real security concern if that container is ever compromised, and part of why bind mounts are common in development and far less common in production.

tmpfs mounts a filesystem that only ever lives in memory, never touching disk, and vanishes completely the instant the container stops. docker run --tmpfs /app/tmp. Reach for it when you genuinely don't want data persisted anywhere, like a decrypted secret you only need for the lifetime of one process.

Docker Compose

A single Dockerfile builds one image. Real applications are rarely one container, so Compose exists to define, network, and start several containers together with one command, all described in a YAML file.

services:
  web:
    build: .
    ports:
      - "8000:8000"
    depends_on:
      db:
        condition: service_healthy
    environment:
      - DB_HOST=db

  db:
    image: postgres:16
    environment:
      - POSTGRES_PASSWORD=devpass
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 5

volumes:
  pgdata:

docker compose up reads this file, creates a shared user-defined network automatically, builds or pulls whatever images are needed, and starts every service, in dependency order where you've declared it. docker compose down tears the whole thing back down. Worth knowing as current: the standalone docker-compose Python binary (with the hyphen) has been end-of-life since 2021, and what you actually run today is docker compose (no hyphen), a plugin built into the Docker CLI itself. If you see a tutorial using the hyphenated form, it's outdated; both still work on most installs for backward compatibility, but the space-separated version is the one you'll actually see in current documentation and in any interview that's testing whether you're current.

The depends_on: condition: service_healthy pattern in the example above is worth understanding, not just copying. A plain depends_on: db only waits for the db container to start, not for Postgres inside it to actually be ready to accept connections, which is a very common cause of a container that keeps crash-looping on startup because it tried to connect to a database that technically existed but wasn't listening yet. Pairing a healthcheck on the dependency with condition: service_healthy on the dependent service fixes exactly that: web won't start until Docker has actually confirmed db is healthy, not just running.

A few Compose commands worth knowing by name, since they come up constantly in real work and in interviews: docker compose up -d starts everything in the background, docker compose down stops and removes containers and the network (add -v to also drop volumes), docker compose build rebuilds images without starting anything, and docker compose logs -f service_name tails logs from just one service instead of the whole stack.

One more feature worth a mention: Compose profiles let you tag certain services so they only start when you explicitly ask for them, useful for things like debugging tools or a monitoring stack you don't want cluttering a normal docker compose up.

services:
  debug-tools:
    image: my-debug-image
    profiles: ["debug"]

docker compose up skips it entirely; docker compose --profile debug up includes it. Handy for keeping one Compose file instead of maintaining several near-duplicate ones.

Security basics

Security questions have become more common in Docker interviews, not just for DevOps-flavoured roles, and the expectations are fairly specific.

Run as a non-root user. By default, whatever runs inside a container runs as root, and root inside a container that escapes its isolation is root on the host. It's a small addition to a Dockerfile:

RUN adduser --system --group appuser
USER appuser

Use a read-only filesystem where you can. docker run --read-only makes the container's entire filesystem read-only except for anything you've explicitly mounted as a volume or tmpfs. If your app has no legitimate reason to write to its own filesystem, this closes off an entire class of attack where a compromised process tries to write and execute something malicious inside the container.

Keep secrets out of environment variables and image layers. Environment variables are convenient but not secure: they show up in docker inspect output and often end up in logs. And anything baked into an image layer at build time, including via ARG, is recoverable from docker history even if you delete it in a later layer, since earlier layers are still there underneath. For a Swarm setup, Docker Secrets encrypts secrets at rest and mounts them as files under /run/secrets/ only inside containers that explicitly request them. Outside Swarm, most teams reach for a dedicated secrets manager: HashiCorp Vault, AWS Secrets Manager, or similar.

Scan your images before they ship. Tools like Trivy and Snyk check your base image and dependencies against known CVE databases, and wiring a scan step into CI catches a vulnerable base image before it ever reaches production instead of after.

Never run latest in production, for the same rollback reason covered in the tagging section earlier, and it's worth repeating here because it's as much a security practice as a hygiene one: you want to know, with certainty, exactly which image is running where.

One more thing worth a mention if the role leans toward platform or security: rootless Docker runs the daemon itself as a non-root user on the host, so even if something escapes a container, it only has the privileges of an unprivileged host user, not host root. It comes with some tradeoffs (a handful of networking and storage features aren't available), but it's increasingly the recommended default for security-conscious setups, especially shared CI runners.

Where Docker stops being enough

A single Docker host works fine until you need more than one, and that's where orchestration comes in. Docker Swarm is Docker's own built-in orchestrator: turn a group of Docker hosts into a cluster, and Swarm handles basic load balancing, service placement, and restarting failed containers. It's genuinely simple to set up, since it's already part of the Docker CLI.

Kubernetes is a separate, far more capable orchestration platform: sophisticated auto-scaling, fine-grained scheduling, self-healing, rolling updates, and support for deployments at a scale Swarm was never really built for. The CNCF's 2025 annual survey put Kubernetes adoption in production at 82% of container users, up from 66% just two years earlier, and that gap keeps widening rather than closing. For most SDE interviews, especially junior and mid-level ones, you're not expected to know Kubernetes deeply unless the job description specifically calls for it. What you are expected to know is the shape of the tradeoff: Swarm is simpler and genuinely fine for smaller deployments, Kubernetes is what nearly everyone reaches for once things get big enough to need serious orchestration, and being able to say that clearly, with the reasoning behind it, covers this topic completely for the vast majority of interviews.

The debugging playbook

This is where interviews increasingly go: not "define a Dockerfile instruction," but "here's a container doing something wrong, walk me through how you'd figure out why." The specific scenario changes, but a consistent method covers nearly all of them: logs first, then network, then a shell, then metrics, in that order, because each step is cheaper and faster than the next, and most problems resolve before you reach the expensive ones.

A container is running but not responding to requests. Start with docker logs --tail 50 -f name. If the application never actually started, or crashed and restarted, the logs usually say so immediately. If the app looks fine in the logs, check whether the port is actually published: docker inspect name and look at the port bindings, since a container that's healthy internally but never had -p set is invisible from outside no matter how correctly it's running. If it still looks fine, get a shell inside with docker exec -it name sh and check from there directly. If you've configured a HEALTHCHECK, docker inspect --format='{{.State.Health.Status}}' name gives you Docker's own verdict without any guesswork.

One container can't reach another. This is a networking problem almost every time, not an application bug, and it's worth saying that out loud in an interview because it shows you're not about to go digging through application code for a routing issue. First, confirm both containers are actually on the same network with docker network inspect network_name, since containers on different networks simply can't see each other by default. Second, and this catches almost everyone at least once: confirm you're connecting using the container name (or Compose service name) as the hostname, not localhost, since localhost inside a container always means that container, never a different one. Third, confirm the target is actually listening on the port you expect from inside that container, using ss -tlnp (the modern replacement for netstat, which most slim images don't even include anymore). If the image is minimal enough that even ss isn't there, check from the host side instead with docker port container_name.

COPY fails during a build even though the file is right there in the repo. Check .dockerignore first, it's the single most common cause by a wide margin: the file is present in your repo but explicitly excluded from the build context. Second, remember the build context is whatever directory you pointed docker build at, not necessarily wherever the Dockerfile itself lives, so if you're running the build from a parent directory with --file path/to/Dockerfile, your context is that parent directory, and any path in COPY is relative to it, not to the Dockerfile. Third, check the exact filename: Linux is case-sensitive, so Data.csv and data.csv are genuinely different files, a mismatch that's invisible on a case-insensitive filesystem like macOS's default but breaks immediately inside a Linux container.

A container keeps restarting in production. docker ps -a first, to see the exit code of whatever's crash-looping, then docker logs --tail 100 name. Three causes cover almost every real case. Exit code 137 means the OOM killer stepped in because the container hit its memory limit, confirm with docker stats and raise the limit if the workload genuinely needs it. A dependency not being ready, like a database the app tries to connect to before it's actually accepting connections, is fixed with the depends_on: condition: service_healthy pattern from the Compose section. And a genuine application crash, a bad input, an unhandled null, a schema mismatch, will usually be sitting plainly in the logs once you look, and needs an actual code fix rather than an infrastructure one. The useful distinction to name out loud in an interview: infrastructure failures (memory, timing, dependencies) and application failures (bugs, bad data) get diagnosed differently and fixed in completely different places, and being able to tell which kind you're looking at quickly is most of what "debugging skill" means here.

Using Docker in a CI/CD pipeline deserves its own mention, since it's a near-guaranteed follow-up once a Docker conversation gets this far. The pattern that comes up over and over: build an image once, tag it with the commit SHA, run your test suite inside that exact image rather than in some separate CI environment, and only push to your registry once tests pass.

# GitHub Actions, roughly
- name: Build image
  run: docker build -t myapp:${{ github.sha }} .

- name: Run tests inside the built image
  run: docker run --rm myapp:${{ github.sha }} pytest tests/

- name: Push to registry
  run: |
    docker tag myapp:${{ github.sha }} ghcr.io/org/myapp:${{ github.sha }}
    docker push ghcr.io/org/myapp:${{ github.sha }}

The point of testing inside the image you're about to ship, rather than in a separate CI runner environment, is that you're validating the actual artifact that goes to production, not something that merely resembles it. Tagging with the commit SHA the whole way through means you can trace, at any point later, exactly which commit is running in any given environment, and a "what changed between this deploy and the last one" question has a one-line answer instead of an investigation.

Redis for SDE interviews: the guide I wish I'd had

Sarthak Rawat — Mon, 21 Sep 2026 00:12:26 +0000

How it works, what to say when an interviewer asks about it, and where it falls over.

Redis comes up in a strange number of interviews. Sometimes it's a direct question, like "what happens when a key expires?" More often it hides inside a system design round. You draw a cache box, the interviewer points at it, and asks what happens when it goes down.

The trouble with learning it is that most material is either a five-minute "Redis is an in-memory key-value store" intro or a reference manual. Neither is what you need. You need the middle: enough depth to explain why things work the way they do, and enough awareness to notice when an interviewer is walking you toward a trap.

This post is my attempt at that middle. It goes through how Redis works, the data types, expiry and eviction, caching patterns, persistence, replication and clustering, transactions and Lua, Pub/Sub, and the system design problems where Redis actually earns its place (locks, rate limiters, leaderboards, queues). It ends with when not to use Redis, which gets asked more often than you'd think. The details match Redis 7 and 8.

What Redis is and how it works

Redis stands for Remote Dictionary Server. It's an in-memory data structure store written in C. The words "data structure" matter more than "in-memory." A plain key-value store maps a key to a blob. In Redis the key is a string and the value is a real data structure: a list, a set, a sorted set, a hash. When you create a sorted set, it simply lives as the value under whatever key you picked. Nearly everything else about Redis follows from that.

People use it as a cache, a session store, a rate limiter, a leaderboard, a lightweight message broker, and sometimes as the main database for data they can afford to lose or rebuild.

Here's what happens when a client sends a command:

The client opens a TCP connection to the server (port 6379 by default) and sends the command using a simple protocol called RESP. It's close enough to plain text that you can type commands into redis-cli and see exactly what goes over the wire.
The server runs an event loop built on epoll (Linux) or kqueue (BSD and macOS). One thread watches thousands of connections at once without needing a thread per client.
When a command is ready, the main thread parses it, runs it against the in-memory data, and writes the reply.
The next command starts only after that one finishes.

That last point is the heart of Redis. Commands run one at a time, on a single thread, in the order they arrive.

A quick note on threads, because interviewers like to poke at "Redis is single-threaded." It's true for command execution. Since Redis 6 you can enable extra I/O threads that handle reading and writing network data. Background threads handle things like fsync for the append-only file and freeing large objects. Snapshots are written by a forked child process. But the commands themselves still run one after another on the main thread. To use more CPU cores, you run more Redis instances, which is what Redis Cluster does.

Why it's fast

Data lives in RAM, so there are no disk seeks on the read path. Running commands on one thread means no locks and no context switching inside the server, and it also means every single command is atomic for free. The event loop lets one thread serve many clients. And the operations themselves are cheap, mostly O(1) or O(log N).

A single node handles on the order of 100k operations per second. The command takes microseconds to run, and over a network you'll see sub-millisecond replies. That speed makes some habits that are terrible against SQL survivable here. Sending 100 small queries in a loop would hurt a database badly. Against Redis, each one is cheap, and you can batch them with pipelining (more on that later) to pay for one network round trip instead of 100.

What that design costs you

Everything has to fit in memory, and memory is the most expensive place to store data. One slow command blocks every other client, which is why KEYS * on a big database is a well-known way to take down production. Durability is weaker than in a relational database (the persistence section covers this). And there are no joins or ad-hoc queries. You decide how you'll read the data first, and design your keys around that.

Two comparisons you'll get asked

Redis vs Memcached: Memcached is a simpler cache. It's multi-threaded and stores plain strings. Redis adds rich data types, persistence, replication, Lua scripting and Pub/Sub. If all you need is a plain cache, either works. Once you want a sorted set or an atomic counter, Redis wins.

Redis vs a SQL database: they're rarely alternatives. Redis trades query flexibility and strong durability for speed and simplicity, so it usually sits in front of or next to a database instead of replacing one.

Data types and what's under them

Interviewers use data types as a shortcut for "does this person understand Redis or have they only used GET and SET?" You should know each type, its main commands, its cost, and a real use for it.

Strings are the base type. A string is binary-safe and can hold up to 512 MB, though you'll almost never go near that. SET and GET are the obvious commands. SET also takes options: EX or PX for a TTL, NX to set only if the key doesn't exist, XX to set only if it does. INCR and DECR treat the string as an integer and change it atomically, which is why counters are trivially safe in Redis. Cache values are usually strings, often JSON.

Hashes are a map of fields to values under one key, so they fit objects nicely. HSET user:42 name Asha city Pune creates one, HGET reads a field, HINCRBY bumps a numeric field. The classic interview question is hash versus a JSON string for an object. With a JSON string you read and rewrite the whole thing to change one field. With a hash you can update one field directly, and it's usually more memory-friendly for small objects. A JSON string is simpler if you always read and write the whole object.

Lists are ordered sequences, implemented as a linked list of compact blocks (called a quicklist). Pushing and popping at either end is O(1), while reaching into the middle is O(N). LPUSH, RPUSH, LPOP, RPOP do what you'd expect, and BRPOP blocks until an item arrives, which is how people built simple job queues before Streams existed. LRANGE reads a slice.

Sets hold unique, unordered strings. SADD, SISMEMBER, SCARD and SMEMBERS are the basics, all cheap except SMEMBERS on a huge set. The interesting commands are SINTER, SUNION and SDIFF, which give you things like mutual friends or users who share tags. Sets suit exact unique tracking, like "which users have already seen this notification."

Sorted sets are the type that comes up most in system design. Each member is unique and has a floating-point score, and the set stays ordered by score. Inside, Redis combines a skip list with a hash table. The skip list gives ordered traversal and rank lookups in O(log N), and the hash table gives O(1) lookup of a member's score. ZADD adds or updates a member, ZINCRBY bumps a score, ZRANGE (with REV for descending order) reads by rank, ZRANK and ZREVRANK give a member's position, and ZRANGEBYSCORE-style queries read by score. Leaderboards, rate limiters, priority queues and "delayed job" schedulers are all sorted sets in disguise.

Streams are an append-only log. Every entry gets an ID (a millisecond timestamp plus a sequence number) and holds field-value pairs. XADD appends, XREAD reads, and consumer groups (XREADGROUP, XACK) let several workers share a stream with acknowledgements. We'll come back to these in the queue section.

Bitmaps aren't a separate type. They're bit operations (SETBIT, GETBIT, BITCOUNT, BITOP) on a string. If you use a user ID as the bit offset, one bit per user per day costs about 12.5 MB for 100 million users. That's how you track daily active users cheaply.

HyperLogLog counts unique items approximately. PFADD adds an item, PFCOUNT returns the estimated number of distinct items. It uses about 12 KB per key no matter how many items you add, with a standard error around 0.8%. The catch is that you can't list what's inside or remove things. If an interviewer asks "how do you count unique visitors across billions of events," this is the answer, along with an honest note about the error.

Geo commands (GEOADD, GEOSEARCH) store coordinates and find points within a radius or box. Under the hood it's a sorted set where the score is a geohash of the coordinates. The search grabs candidates from grid-aligned boxes first, then filters to the exact radius.

Redis 8 also ships probabilistic structures like Bloom filters, plus JSON and time series support, in the core. On older versions these came from separate Redis Stack modules. A Bloom filter says either "definitely not in the set" or "probably in the set," never the other way round, and it will matter when we get to cache penetration.

One habit that's worth building early: name keys like user:42:profile or order:9001:items. Colons are just a convention, but every team uses them, and your key design is how you'll shard the data later.

Expiration and eviction

People mix these two up constantly, so keep them apart in your head. Expiration is about staleness: a key has a time to live and disappears when it runs out. Eviction is about memory: when Redis is full, it throws away keys to make room. They're separate mechanisms with separate settings.

Expiration. You set a TTL with EXPIRE key 60, or directly in SET key value EX 60. TTL tells you how many seconds remain, and PERSIST removes the TTL. One gotcha: a plain SET on an existing key wipes its TTL unless you pass KEEPTTL. That has caused real bugs.

Redis removes expired keys in two ways. When a client touches an expired key, Redis notices and deletes it on the spot (lazy expiration). Separately, a background task samples keys that have TTLs several times a second and removes the expired ones. So an expired key can sit in memory for a short while, but you'll never read it. Redis guarantees that you won't see a value after its TTL has passed.

Eviction. By default Redis has no memory limit on a 64-bit machine. You set one with maxmemory 2gb, and you choose what happens at the limit with maxmemory-policy. The default policy is noeviction, which means Redis refuses new writes once it's full. That surprises people who assumed a cache would just make room. For a cache, you pick something else:

allkeys-lru evicts the least recently used keys from the whole keyspace. It's a good default when some keys are much more popular than others, which is true of most workloads.
allkeys-lfu evicts the least frequently used keys. It's better when a key that was popular for a long time shouldn't be dropped just because nobody touched it in the last minute.
allkeys-random evicts random keys. It suits workloads that scan through everything evenly.
The volatile- versions (volatile-lru, volatile-lfu, volatile-random) only consider keys that have a TTL. volatile-ttl evicts keys with the least time left.

If no keys have a TTL, the volatile- policies behave exactly like noeviction. So if you pick one and forget to set TTLs, your cache will start rejecting writes. Newer versions also add "least recently modified" policies, but the ones above cover what interviews ask.

Redis's LRU is an approximation. It doesn't track the exact order of every key, which would cost memory. Instead it samples a handful of keys (the maxmemory-samples setting, 5 by default) and evicts the best candidate among them. LFU works similarly, using a small counter per key that decays over time.

A practical detail: eviction happens when a command that would add data arrives while memory is over the limit. And Redis needs headroom beyond maxmemory, for replication buffers, fragmentation, and the extra pages copied while a child process is writing a snapshot. Setting maxmemory to 100% of the machine's RAM is a common way to get the process killed by the operating system.

If you're asked "cache is full, what happens?" the answer is: it depends on maxmemory-policy, and the default rejects writes.

Caching patterns and how they fail

Caching is the most common reason Redis is in a diagram, and the part interviewers dig into.

The four patterns.

Cache-aside is the default. The application checks Redis first. On a miss it reads the database, writes the result to Redis (with a TTL), and returns it. The app talks to both systems, and the cache only ever holds data someone asked for.

Read-through looks the same from the outside, but the cache layer loads the data itself on a miss and the app only talks to the cache. Redis has no built-in loader, so read-through means a library or a small service that wraps Redis and your database.

Write-through sends every write to the cache and the database together. Reads stay fresh, writes cost a bit more.

Write-behind (also called write-back) writes to the cache and flushes to the database later, asynchronously. Writes are fast, but if Redis dies before the flush, those writes are gone, and you have to handle ordering and retries.

Invalidation. The three ways to keep cached data from going stale are a TTL, deleting the key when the underlying data changes, and write-through. Most systems use a TTL as a safety net and also delete on write.

On writes, deleting the cached key is usually safer than updating it. Two writers racing to update the cache can leave the older value winning. Even deleting has a race: a reader misses, reads the old row from the database, and gets delayed. Meanwhile a writer updates the database and deletes the key. Then the slow reader finally writes the old value into the cache. Now the cache is stale until the TTL runs out. It's rare, but it's why you always want a TTL, and why some teams delete the key a second time after a short delay.

The cache is also eventually consistent by nature. Reads from a replica can lag behind the primary too. If your product can't tolerate any staleness, say so in the interview and explain how you'd narrow it.

The three failure modes. These names are used inconsistently across blog posts. One popular article uses "penetration" for the hot-key case, while most sources use it for missing keys. Define the term in your own words before you answer, and you'll be fine.

Cache penetration is when lots of requests ask for keys that don't exist anywhere, so they miss the cache every time and hit the database every time. Someone scanning random IDs does this, deliberately or not. There are two standard fixes. You can cache the "not found" answer for a short time (a null value with a small TTL). Or you can put a Bloom filter in front holding every valid key, and reject requests that the filter says are definitely absent. The filter can have false positives, which just means an occasional wasted lookup.

Cache stampede (also called the thundering herd, or "breakdown") is when one very popular key expires and, in the split second before it's refilled, hundreds or thousands of requests all miss and all hit the database to rebuild the same value. Three fixes come up. The first is a mutex: the first request to miss grabs a lock with SET lock:key token NX PX 5000 and rebuilds the value, while the others wait a moment and retry, or serve a slightly stale copy. The second is probabilistic early expiration (sometimes called X-Fetch), where each request has a small, growing chance of refreshing the value shortly before it actually expires, so one request quietly refreshes it while everyone else keeps reading the old one. The third is never letting the hottest keys expire at all, and refreshing them from a background job.

Cache avalanche is many keys expiring at once, or the cache itself going down. If you filled the cache at 9:00 with a 60 minute TTL on everything, then at 10:00 the database gets the entire load at once. The fix is jitter: add a random amount to each TTL so expiry spreads out. For the "cache is down" case, you need a plan too: rate limit or shed load in front of the database, and maybe replicate Redis so one failure doesn't empty the cache.

Hot keys. Sometimes traffic doesn't spread evenly. Imagine a cluster of 100 nodes caching product data, and one product goes viral. The single node holding that key takes as much traffic as the other 99 combined. Adding nodes doesn't help, because the key lives on exactly one of them.

There are three usual remedies, each with a cost. A small in-process cache on each app server absorbs most reads for the hottest keys, at the price of data that can be stale for as long as that local cache's TTL. Key copies store the same value under several names (product:123:1 through product:123:10), which hash to different nodes, and readers pick one at random. Writes then have to update every copy. Read replicas add read capacity, but only if your clients are set up to read from replicas, and they do nothing for a key that's hot for writes.

Spotting a possible hot key in your own design, before the interviewer asks, is the kind of thing that stands out.

Persistence

Redis lives in memory, but it can write to disk so it survives a restart. There are two mechanisms, and you can use them together or turn both off.

RDB snapshots. Redis writes the whole dataset to a compact file (dump.rdb) at intervals. A config line like save 60 1000 means "snapshot if at least 1000 keys changed in the last 60 seconds." You can also trigger it with BGSAVE. To do this without pausing, Redis forks: the child process writes the file while the parent keeps serving clients, and copy-on-write means memory is only duplicated for pages that change during the snapshot.

RDB files are small, great for backups, and load fast on restart. The downsides are that you lose everything since the last snapshot on a crash (often minutes), and fork() on a very large dataset can pause Redis for a noticeable moment.

AOF (append-only file). Redis logs every write command, and on restart it replays the log to rebuild the data. AOF is off by default (appendonly yes turns it on). How much you can lose depends on the appendfsync setting:

always flushes to disk on every write batch. It's the safest and by far the slowest.
everysec flushes once a second, and it's the default. You can lose up to about a second of writes.
no leaves flushing to the operating system, which is fast and least safe.

The log grows forever if you leave it, so Redis periodically rewrites it in the background into the shortest list of commands that produces the current data. Since Redis 7 the AOF is split into a base file plus incremental files, tracked by a manifest.

Which should you use? Redis's own guidance is to use both if you want durability comparable to a traditional database. If you can live with a few minutes of loss, RDB alone is fine. For a pure cache, you can use neither. If both are on and Redis restarts, it loads the AOF, because that's the most complete record.

The honest answer to "is Redis durable?" is "only as durable as you configure it, and never quite like a database that flushes every commit." With the default everysec, an acknowledged write can still be lost. And replication adds another window on top, which brings us to the next section.

Replication, high availability and scaling

One Redis node has two limits: if it dies you lose availability, and it can only hold as much as one machine's memory. Replication, Sentinel and Cluster each deal with a piece of that.

Replication. You point a replica at a primary (REPLICAOF host port). The replica does a full sync first: the primary produces an RDB snapshot and sends it over, followed by the writes that piled up during the transfer. After that the primary streams every write command to the replica. If the connection drops briefly, the replica reconnects and asks to continue from its last offset. If that offset is still in the primary's replication backlog (a fixed-size buffer), only the missing part is sent (a partial resync). Otherwise it's another full sync.

The thing to remember is that replication is asynchronous. The primary acknowledges your write before the replica has seen it.

So if the primary dies right after acknowledging a write, and a replica that hasn't received it gets promoted, that write is gone. This is the core reason Redis isn't a system of record, and it shows up again in the distributed lock section. The WAIT command makes a client block until N replicas confirm a write, which shrinks the window but doesn't eliminate it.

Replicas are read-only by default. You can spread reads across them, but every replica read can be a little behind the primary. That's fine for a product page and wrong for something like "did my payment go through."

Sentinel. Replication alone doesn't promote anyone when a primary dies. Sentinel is a set of separate processes that watch the primary and replicas and handle failover. You run at least three, on different machines. When a Sentinel can't reach the primary, it marks it as "subjectively down." Once enough Sentinels agree (the quorum you configured), it's "objectively down." The Sentinels then elect a leader, and that leader picks the best replica (by priority and how much data it has) and promotes it, and reconfigures the others. Clients ask Sentinel "who is the current primary?" instead of hardcoding an address.

Sentinel gives you high availability, and nothing more. All your data still lives on one primary, so it doesn't add capacity. It also can't bring back writes lost to asynchronous replication.

Cluster. When the data no longer fits on one machine, or one machine can't keep up with the traffic, you shard it, and Redis Cluster is the built-in way.

The keyspace is divided into 16,384 hash slots. A key's slot is CRC16(key) mod 16384. Each primary owns a range of slots, and each primary usually has one or more replicas for failover. Nodes exchange information with each other using a gossip protocol, so every node knows the full slot-to-node map.

Clients cache the slot map and send each command straight to the right node. If a slot has moved, the node answers with a MOVED error that includes the correct address, and the client refreshes its map. During a resharding operation you might see ASK, which is a one-time redirect for a slot that's midway through moving. Nodes don't forward requests for you, and there's no router that splits a request across nodes and merges the results.

That last point is what makes Cluster feel restrictive. Commands that touch several keys (MGET, SINTER, MULTI transactions, Lua scripts) only work when every key is in the same slot. To force that, you use hash tags: only the part of the key inside curly braces is hashed. {user:42}:posts and {user:42}:likes always land in the same slot. The takeaway is that with Cluster, how you name your keys is how you scale.

A few more things worth knowing. Failover in Cluster works without Sentinel: replicas get promoted when a majority of primaries agree a primary has failed. You need at least three primaries to have a majority. Cluster only supports database 0. And during a network partition, a primary stuck on the minority side can keep accepting writes for a short time before it stops, and those writes can be lost.

Interviewers sometimes ask how hash slots differ from consistent hashing. With consistent hashing, nodes and keys sit on a ring and a key belongs to the next node clockwise, so adding a node automatically takes over part of a neighbour's range. Redis uses a fixed number of slots and an explicit table of who owns which. Moving data means moving specific slots, which gives operators direct control, at the cost of that table being something the cluster has to keep consistent.

To choose between them: if everything fits on one machine and you want failover, primary plus replicas with Sentinel is enough (or a managed service that does the same). If you need more memory or write throughput than one node offers, you need Cluster.

Atomicity: transactions, Lua and pipelining

Because commands run one at a time, every single command is atomic. INCR can't lose an update the way GET followed by SET can, because two clients doing GET then SET can both read 5 and both write 6. That's why INCR exists.

The trouble starts when you need several steps to happen together. Redis has three tools that get confused with each other.

Transactions with MULTI and EXEC. MULTI starts queuing commands, and EXEC runs the queue. Nothing from any other client runs in the middle of it. But there's no rollback. If one command fails at runtime (say, INCR on a key holding text), the others still run. If a command is malformed and fails while being queued, the whole transaction is refused at EXEC. That's very different from a SQL transaction, and interviewers like to check that you know it.

For optimistic concurrency, use WATCH key before MULTI. If the watched key changes before EXEC, the transaction aborts, and you retry. It's the Redis version of compare-and-swap.

Lua scripts. EVAL sends a script to the server, and the whole script runs as a single command. Nothing else interleaves, and you can read a value, make a decision, and write, all atomically. That's the thing MULTI can't do, because inside MULTI you can't use an earlier result to decide a later command. Lua is how you build a safe lock release or a rate limiter. Two cautions: a long script blocks everybody, and in Cluster you must pass every key the script touches as a key argument so Redis can check they share a slot.

Pipelining. A pipeline just sends many commands without waiting for each reply, then reads all the replies. It saves network round trips and can speed things up dramatically. It is not atomic. Other clients' commands can slip in between yours.

Pub/Sub

Pub/Sub is a messaging pattern where publishers send messages to a named channel and every client subscribed to that channel at that moment receives them. Publishers and subscribers don't know each other exist.

The commands are short: SUBSCRIBE news listens on a channel, PSUBSCRIBE news:* listens on a pattern, and PUBLISH news "hello" sends a message and returns how many subscribers received it. A channel needs no setup. It exists as long as someone is subscribed.

The one property you must state clearly is that delivery is at-most-once and nothing is stored. If a subscriber is offline, or its connection blips, it never sees the message. There are no acknowledgements, no replay and no history. A subscriber that can't keep up gets disconnected once its output buffer passes a limit, so it doesn't eat the server's memory. And a client that's subscribed can't run normal commands on that connection under the older protocol, so apps use a separate connection for their other work.

There's a common outdated claim about scaling. In classic cluster Pub/Sub, every message was broadcast to every node, so adding nodes didn't add capacity. Since Redis 7 there's sharded Pub/Sub (SPUBLISH and SSUBSCRIBE), where a channel is hashed to a slot like a key, and only that shard handles it. Capacity grows with the cluster. Connection cost is per node, not per channel, so millions of channels doesn't mean millions of connections.

How does it compare with the alternatives? Streams keep messages, let consumers catch up after downtime, and support acknowledgements. They give at-least-once delivery. Kafka does the same at much larger scale with long retention and replay for many independent consumers. Pub/Sub is the right tool when you only care about whoever is listening right now and losing a message is acceptable.

In system design, that usually means something like chat across many app servers. Each server subscribes to the channels its connected users care about. When someone sends a message, it's published once, every server with an interested user receives it, and pushes it down the WebSocket. Live dashboards and presence indicators work the same way. If the requirement says offline users must receive it later, Pub/Sub alone isn't enough. Write the message to a database or a stream too.

A related trap: keyspace notifications (getting an event when a key expires, for example) are delivered over Pub/Sub, so they inherit the same lack of guarantees. Don't build correctness-critical logic on them.

System design problems where Redis shows up

This is where everything above gets used. For each one, say which data structure you'd pick and why, then say what could go wrong.

A distributed lock. Say several app servers must not do the same thing at once, like two people booking the same concert seat. Redis works as a lock manager because it's one shared place every server can reach, and each command is atomic. The lock is just a key:

SET lock:seat:343 <random-token> NX PX 30000

NX means "only set it if it doesn't exist," so exactly one caller gets OK. PX 30000 gives the lock a 30 second lease, so a crashed process can't hold it forever. The random token identifies who owns it.

Releasing is where people slip. If you just DEL the key, you might delete someone else's lock, because yours may have expired while you were still working and another client took it. So you delete only if the token still matches, and the check and delete have to be atomic, which means a Lua script:

if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
else
  return 0
end

Now the part interviewers actually care about, which is why this isn't a real guarantee. If Client A stalls (a long garbage collection pause, a slow network) past the lease, the lock expires, Client B takes it, and then Client A wakes up and carries on, believing it still holds the lock. Both write. A second problem is replication: if the primary grants the lock and dies before the replica hears about it, the promoted replica happily grants the same lock again.

The Redlock algorithm tries to solve the second problem by acquiring the lock on a majority of independent Redis nodes. It's controversial. Martin Kleppmann wrote a well-known critique showing that it still can't survive the paused-client problem. The standard defence for that is a fencing token: each lock grant comes with an increasing number, and the storage layer rejects any write carrying an older number than one it has already seen. Redis doesn't give you that out of the box.

So a Redis lock is good for efficiency (avoiding duplicate work most of the time) and shaky for correctness. If a stale lock holder would corrupt data, enforce the rule where the data lives. A SELECT ... FOR UPDATE row lock or a conditional update like UPDATE ... WHERE version = 7 in your database often removes the need for a distributed lock entirely. Saying this out loud in an interview scores well.

A cousin of the lock, for things like ticket booking, is a temporary hold: SET seat:343:A12 user42 NX EX 600 reserves a seat for ten minutes, and the TTL releases it if the user walks away.

A rate limiter. The simplest version is a fixed window. For each user and time window you keep a counter: run INCR on a key like rl:user42:1700000000, and if the result goes over the limit, reject with a 429 and a Retry-After header. One subtlety trips people up. You want to set the expiry only when INCR returns 1, meaning the first request in the window. If you call EXPIRE on every request, steady traffic keeps pushing the expiry forward and the window never resets. And if the process crashes between INCR and EXPIRE, you've got a counter that never expires. So do both steps in one Lua script:

local current = redis.call("INCR", KEYS[1])
if current == 1 then
  redis.call("EXPIRE", KEYS[1], ARGV[1])
end
return current

The weakness of a fixed window is bursts at the edges. A user can send the full limit at the very end of one window and again at the start of the next, doubling the rate for a moment.

A sliding window fixes that with a sorted set per user, where each request is a member and its timestamp is the score. On each request you drop entries older than the window, count what's left, and add the new one if the count is under the limit. Again, do it in one script so it's atomic:

-- KEYS[1] = key, ARGV[1] = now (ms), ARGV[2] = window (ms)
-- ARGV[3] = limit, ARGV[4] = a unique id for this request
redis.call("ZREMRANGEBYSCORE", KEYS[1], 0, ARGV[1] - ARGV[2])
if redis.call("ZCARD", KEYS[1]) < tonumber(ARGV[3]) then
  redis.call("ZADD", KEYS[1], ARGV[1], ARGV[4])
  redis.call("PEXPIRE", KEYS[1], ARGV[2])
  return 1
end
return 0

Use a unique member for each request. Two requests in the same millisecond would otherwise overwrite each other. The cost is memory: the set holds up to limit entries per user. If that's too much, mention the token bucket or a sliding window counter as cheaper approximations, and be ready to explain the trade-off between accuracy and memory.

A leaderboard. This is a sorted set, and it's about the easiest system design win there is. ZADD board 1500 alice sets a score (and re-adding the same member just updates it), ZINCRBY board 10 alice adds points, ZREVRANGE board 0 9 WITHSCORES returns the top ten, and ZREVRANK board alice gives a player's rank. Every operation is O(log N) plus the size of the slice you read. To keep only the top N, trim with ZREMRANGEBYRANK board 0 -101, which removes everything below the top 100. Ties are broken by member name, so if ties matter, say how you'd handle them. If someone pushes you to a huge scale, note that one sorted set is one key on one node, so you'd split it by region or by score range and merge for global queries.

A job queue. Lists with LPUSH and BRPOP work, but if a worker pops a job and then dies, the job is gone. Streams with consumer groups fix that:

XADD jobs * type email to a@b.com
XGROUP CREATE jobs workers 0 MKSTREAM
XREADGROUP GROUP workers worker-1 COUNT 10 BLOCK 5000 STREAMS jobs >
XACK jobs workers 1700000000000-0
XAUTOCLAIM jobs workers worker-2 60000 0-0

A producer appends with XADD. Workers in a group read with XREADGROUP, and Redis tracks each delivered but unacknowledged entry as pending for that worker. When the worker finishes, it calls XACK. If a worker dies, its pending entries sit idle, and another worker claims them with XAUTOCLAIM after they've been idle long enough (60 seconds in the example).

The catch: this gives you at-least-once processing. Redis can't tell a slow worker from a dead one, so a job can occasionally run twice. Make your jobs idempotent. Also, stream entries are only as durable as your persistence settings, so with defaults a crash can lose recent ones. Streams fit modest queues where you already run Redis, like background jobs, notification fan-out and work distribution. Kafka is the better answer when you need long retention, replay for many independent consumers, or throughput where losing a message is unacceptable.

A session store. Store each session as a hash (or a JSON string) under session:<id> with a TTL, and refresh the TTL on each request so active users stay logged in. Logging out is a DEL. It's fast and the TTL cleans up abandoned sessions. The question to raise yourself is what happens if Redis restarts and forgets everyone. If that's unacceptable, turn on AOF, or keep sessions in a database and use Redis as a cache in front of them.

Counting things. INCR for exact counters. HyperLogLog (PFADD, PFCOUNT) for approximate unique counts in tiny memory. Bitmaps for per-user yes/no flags across a huge user base, like daily active users, and BITOP AND across days to find users who came back. Geo commands for "nearby" queries.

When not to use Redis

Interviewers ask this to see whether you understand the limits, so have an answer ready.

Don't make it the only copy of data you can't lose. Between asynchronous replication and the persistence loss windows, acknowledged writes can vanish. Don't use it when your working set can't fit in memory at a cost that makes sense. Don't expect query flexibility: there are no joins, no secondary indexes out of the box, and in a cluster, multi-key operations only work within one slot. And don't use it as a replayable event log with long retention for many consumers, which is what Kafka is built for.

A few operational things worth knowing

You won't be asked to be a Redis administrator, but a handful of commands and habits make you sound like someone who has run it.

INFO prints server stats in sections, such as memory, persistence, replication and stats. The keyspace_hits and keyspace_misses counters let you work out your cache hit ratio. SLOWLOG GET shows the slowest recent commands. MONITOR streams every command the server receives, which is handy for debugging and expensive in production, so avoid leaving it running. Use SCAN to walk the keyspace in small steps instead of KEYS, and UNLINK to delete big keys without blocking (DEL frees memory on the main thread). redis-cli --bigkeys and MEMORY USAGE key help you find oversized keys.

When latency spikes, the usual suspects are a slow command on a big collection, a large key being deleted, a fork for a snapshot on a big dataset, or the machine swapping to disk. On security, don't expose Redis to the public internet, set authentication and ACLs, and use TLS if traffic crosses networks you don't control.

One more piece of context, in case it comes up. In 2024 Redis moved away from its permissive BSD license, and the Linux Foundation backed a fork called Valkey, based on Redis 7.2.4 and still BSD-licensed. Redis 8 later added AGPLv3 as a license option. The two projects speak the same protocol, so most clients work with either, but their features are starting to diverge. Managed services like AWS ElastiCache offer Valkey too.

How I'd practise this

Reading only gets you partway. Run Redis in Docker (docker run -p 6379:6379 redis) and type the commands into redis-cli until they feel obvious. Then build the three things interviews keep asking for: a lock with a safe release script, a sliding window rate limiter, and a leaderboard. Once those work, try to break them: kill the process mid-script, let a TTL expire at a bad moment, and see what happens.

Then try answering these out loud, without notes:

Why is Redis fast, and what's the cost of that design?
What's the difference between expiration and eviction, and what does the default policy do when memory is full?
Walk me through cache-aside. What are the stale-data races?
What's the difference between cache penetration, stampede and avalanche, and how do you fix each?
How do RDB and AOF differ, and what would you use for a cache versus for sessions?
A primary fails right after acknowledging a write. What can happen?
When would you use Sentinel, and when Cluster? How does a key find its node?
What's the difference between a pipeline, MULTI/EXEC and a Lua script?
Why doesn't Pub/Sub work for messages that must not be lost, and what would you use instead?
Design a rate limiter. Now design a distributed lock, and tell me why it isn't fully safe.
Give me three reasons not to use Redis as your main database.

If you can answer those in your own words, with an example each, you know Redis well enough for most SDE interviews.

My AI Support Bot Was Slow. Then SigNoz Told Me My Alert Would Fire in 158 Years.

Sarthak Rawat — Tue, 14 Jul 2026 16:18:20 +0000

Author's Note

This project was built as part of the Agents of SigNoz hackathon organized by WeMakeDevs and SigNoz.

Source code: https://github.com/SarthakRawat-1/signoz-assistant

How I traced a real latency problem, a real auth failure, and one genuinely baffling unit-conversion bug in an AI support assistant, using OpenTelemetry and self-hosted SigNoz.

I built a small AI support assistant for an imaginary ecommerce company, the kind that answers "where's my order" and "will the rain delay my delivery." It worked, then I sent it 25 requests back to back and some took over three seconds. No errors, no crashes, just slow, some of the time. I had no idea which part, so I did what I should have done from the start: instrumented it and watched.

Two things came out of that afternoon. One was a real, boring, entirely explainable latency bug. The other was SigNoz telling me an alert would trigger in roughly 158 years, which is not a sentence I expected to type while debugging a chatbot.

What I built

The app is small on purpose: a FastAPI endpoint (POST /support/chat) that takes a message and a conversation_id, stores history in SQLite, optionally calls a weather API if the message looks shipping-related, and generates a reply. A small frontend on top shows the trace ID and classified intent for whatever you just sent, mostly so I didn't have to keep tabbing over to SigNoz to find which trace belonged to which message.

One honest disclosure: I didn't have a Gemini API key wired in for most of this session, so the LLM calls below are running against a stub client, not live Gemini. My code logs this plainly: No GEMINI_API_KEY set. Falling back to Stub LLM Client. It doesn't affect the story, since the point was never to benchmark Gemini, it was to see whether SigNoz would correctly point at the real bottleneck regardless of which piece was mocked. It did.

The weather lookup is where the interesting instrumentation lives, a manual span wrapping both outbound calls with business-relevant attributes attached along the way:

async def get_weather(city: str) -> dict:
    with tracer.start_as_current_span("weather.lookup_operation") as span:
        span.set_attribute("weather.city", city)

        async with httpx.AsyncClient() as client:
            geo_response = await client.get(GEOCODING_URL, params={"name": city, "count": 1})
            span.set_attribute("weather.geocoding.http.status_code", geo_response.status_code)

            lat, lon = _extract_coords(geo_response.json())
            span.set_attributes({"weather.latitude": lat, "weather.longitude": lon})

            forecast_response = await client.get(FORECAST_URL, params={"latitude": lat, "longitude": lon})
            span.set_attribute("weather.forecast.http.status_code", forecast_response.status_code)
            # ...response is parsed into a simplified condition below, trimmed here

The two client.get() calls, one geocoding, one forecast, are the two GET spans stacked under weather.lookup_operation in the waterfall below. Attaching customer.intent and conversation.id to spans like this let me filter traces by business behavior, "show me every shipping_delay conversation," instead of only by HTTP endpoint, which mattered more once several conversations were running at once.

No agent framework, no tool-calling loop, no vector database. Just enough moving parts to produce interesting telemetry.

Getting visibility

I self-hosted SigNoz using Foundry, their newer install method that replaced the old docker-compose-based install a few months back: install the CLI, point it at a small YAML file, and it stands up ClickHouse, the OTel collector, and the UI itself. I added HTTPXClientInstrumentor on top of FastAPI's auto-instrumentation, plus manual spans around the LLM call and the weather lookup.

None of it went smoothly first try. Docker Desktop was running, but its CLI binaries weren't on my shell's PATH, so foundryctl cast failed outright with tools are not available, please install them and try again: docker, docker-compose, a two-minute fix once I found it. Then logs didn't show up in SigNoz at all, because auto-instrumented logging assumes your app and collector share stdout, which isn't true when your app runs on the host and SigNoz runs in Docker. Configuring an OTLP log exporter to ship logs over gRPC fixed it, and correlation with traces worked immediately after.

Investigation #1: the actual slow part

Once data was flowing, I sorted traces by duration and opened the slowest one. Without the waterfall view, I'd probably have blamed Gemini first, since it's the component that looks expensive on paper. The trace showed I was about to optimize the wrong thing entirely: weather.lookup_operation took 3.06 seconds. The instrumented LLM span, right next to it, took 405 milliseconds. This wasn't a fluke. I checked several of the slow traces, and the same pattern held every time.

I built a small dashboard to track this at the aggregate level too: p95 latency on the endpoint, error rate, and average LLM token count.

Across 25 seeded requests, the endpoint's overall p50 was a perfectly reasonable 484ms, but p99 jumped to 3.37 seconds. weather.lookup_operation alone had a p50 of 2.4 seconds. Four requests triggered it; all four were the slow ones.

Neither call is artificially delayed; there's no sleep() anywhere in that path. Two things make it slow: the calls are sequential, since you need coordinates before requesting a forecast, and each request creates a new AsyncClient instead of reusing one, so every lookup pays the cost of a fresh connection twice. Mundane, fixable, and something I'd never have found by reading logs alone; the waterfall made both GET calls and their relative cost obvious at a glance.

Investigation #2: an intentional failure

The second investigation, unlike the first, was deliberate. I temporarily set an invalid Gemini API key to see what a real auth failure would look like end to end.

The result showed up as an ERROR-level log, Request failed with exception, with a trace_id attached directly to the log line. Clicking that ID took me straight to the exact trace where the failure happened, no manual correlation required. This is the kind of thing that's genuinely hard to appreciate until you've done the alternative: grepping through unstructured logs trying to match timestamps to a request you think might be related.

The bug I didn't expect: nanoseconds pretending to be years

This is the one I'd actually lead with if I were telling this story out loud. I set up an alert: notify me if p95 latency on /support/chat goes above 4 seconds. Simple enough. Except when I opened the alert's chart to sanity-check it, the y-axis read 158.44 years, with the critical threshold drawn at 126.76 years.

SigNoz's alert query builder operates on duration_nano, nanoseconds, not milliseconds. I'd set my threshold to 4000, assuming milliseconds. It meant 4000 nanoseconds, roughly the time it takes an electron to get mildly annoyed, so every request blew past it instantly, and the chart rescaled itself into whatever unit made the numbers legible: years. The fix was 4000000000 (4 billion nanoseconds, i.e., 4 seconds). The lesson wasn't "SigNoz is confusing," it was that observability tools expose raw units because precision matters at the storage layer, and it's on you to know exactly what you're alerting on.

With the units corrected, the same alert did exactly what I'd originally wanted. It stayed quiet during normal traffic and fired only once a request genuinely crossed 4 seconds.

I couldn't find this mentioned anywhere in the getting-started docs, and it took me longer than I'd like to admit to realize the alert itself was working correctly and my units weren't.

What actually changed for me

Before this, I thought of observability mostly as dashboards: numbers on a screen telling you something's wrong. What I didn't expect is how much of the actual debugging happens in traces, not metrics. A metric can tell you p99 latency is bad; it can't tell you why. The trace waterfall explained the "why," span by span, in a way I could act on immediately instead of forming a hypothesis and checking it against logs one line at a time.

The other shift is smaller but sticks with me: I used to think of alerts as something you configure once you already understand your system. This one taught me the alert itself needs debugging too, not just the app it's watching.

One thing I appreciated only after the fact: none of this instrumentation is actually SigNoz-specific. The spans and attributes are emitted through OpenTelemetry, a vendor-neutral standard; SigNoz just happens to be the backend consuming them here. That's why it felt like a real part of the application rather than something bolted on for one dashboard.

Conclusion

Before this, "my app is slow" would have meant staring at logs and guessing. After wiring up OpenTelemetry and SigNoz, every request told its own story: which span cost what, which log line belonged to which trace, and, once, which threshold was off by a factor of a million. Next, I'd share one long-lived httpx.AsyncClient instead of opening a new one per request, run the geocoding and forecast calls concurrently, and wire in a real Gemini key just to see what surprises a live model call brings. I didn't get a dramatic engineering war story out of this. I got a small, boring, entirely real latency bug, an honest auth failure, and one legitimately funny unit-conversion mistake. That's a fair trade for an afternoon.

Disaggregated Prefill/Decode: The Architecture Quietly Rewiring How LLMs Get Served

Sarthak Rawat — Wed, 08 Jul 2026 16:59:09 +0000

If you have used any large language model product in the last year, you have run into a piece of infrastructure most people never think about: the serving system that turns a prompt into a stream of tokens. For a long time, that system worked the same way regardless of scale. One GPU, or one small cluster of GPUs, would take your prompt, process it, and then generate the response token by token, all on the same hardware, in the same process.

That approach is quietly breaking down. Context windows have stretched past 100,000 tokens, request volumes have grown by orders of magnitude, and the cost of running inference has become a board level conversation at most AI companies. The fix that has emerged, and is now running in production at some of the largest LLM providers in the world, is called disaggregated prefill/decode serving. It sounds like a narrow infrastructure detail, but it is one of the more interesting distributed systems problems in production today, and it has not been written about nearly as much as caching or load balancing.

This post explains what the problem actually is, how the architecture works, how real systems like Mooncake and NVIDIA Dynamo implement it, and when it is genuinely worth adopting versus when it is just extra operational burden.

Why prefill and decode do not want to live together

Every LLM inference request has two distinct phases.

Prefill is when the model reads your entire prompt and builds up its internal representation of it, layer by layer, for every token you sent in. This step is compute bound. It uses the GPU's raw math throughput heavily, and it happens once per request, regardless of how long the eventual response is.

Decode is what happens after that: the model generates the response one token at a time, each new token depending on everything that came before it. This step is memory bandwidth bound. The GPU spends most of its time moving the key value cache (the KV cache, which stores the model's running memory of the conversation) in and out of memory rather than doing heavy computation.

These two phases have almost opposite hardware profiles. Prefill wants raw FLOPs. Decode wants memory bandwidth and low latency access to the KV cache. When both run on the same GPU in the same batch, they compete for the same resources, and one interferes with the other. A long prompt arriving for prefill can stall the token generation of requests that are already mid decode, which shows up to the end user as a sudden latency spike.

Academic work on this problem, including the DistServe and Splitwise papers, and production systems like Mooncake, converge on the same conclusion: separate the two phases onto different hardware, and let each one be scheduled and scaled independently.

The core architecture

At a high level, disaggregated serving introduces a new component that does not exist in a monolithic setup: a layer that moves the KV cache from the machines doing prefill to the machines doing decode.

The router (sometimes called a scheduler, sometimes a conductor, depending on the system) decides which prefill instance and which decode instance should handle a given request. It is trying to balance several things at once: current load on each pool, whether part of the KV cache for this request already exists somewhere in the cluster from a previous turn, and the latency target for this particular request.

Once a prefill instance finishes processing the prompt, it does not throw away its work. It streams the resulting KV cache, often layer by layer as it is produced rather than waiting for the whole thing to finish, over to the assigned decode instance. The decode instance loads that cache and begins generating tokens, which get streamed back to the client as they are produced.

Here is what a single request actually experiences as it moves through the system.

The hard part: moving the KV cache fast enough

The entire architecture only works if the KV cache transfer is fast and does not become its own bottleneck. If moving the cache takes longer than the time saved by separating the phases, disaggregation is pointless.

This is why the transfer layer has become its own area of serious engineering effort. In 2026, most production systems rely on RDMA (remote direct memory access) over InfiniBand or RoCE networking to move KV cache data directly between GPU memory pools with very low latency, falling back to plain TCP when that hardware is not available. NVIDIA's NIXL and Moonshot AI's Transfer Engine (the foundation of Mooncake) are the two most widely used implementations of this idea, and both vLLM and SGLang have built native support for them.

The amount of data being moved is not trivial either. For a rough sense of scale, the size of the KV cache for a single request scales with the number of layers in the model, the number of attention heads, the size of each head, and the length of the sequence. For a long conversation or a large document being summarized, this can be gigabytes of data that needs to move between machines before the first output token is even generated, which is why the transfer engine, not just the model itself, has become a serious point of optimization.

Case study: Mooncake, the platform behind Kimi

The clearest public example of this architecture running at real scale is Mooncake, the serving platform Moonshot AI built for its Kimi chatbot. Mooncake was one of the first systems to fully commit to a KV cache centric design rather than treating the cache as a side effect of inference.

At the center of Mooncake sits a global scheduler called the Conductor. For every incoming request, the Conductor decides which prefill and decode instance pair should handle it, taking into account how much of the relevant KV cache already exists somewhere in the cluster and can be reused rather than recomputed from scratch. Mooncake also does something unusual: it uses otherwise idle CPU memory, DRAM, and even SSD storage across the GPU cluster to build a much larger, distributed KV cache pool than would fit in GPU memory alone.

The results reported by the team are substantial. In their published research, Mooncake increased effective request capacity by roughly 60 to nearly 500 percent over baseline approaches in various tested scenarios, while still meeting latency service level objectives, with the biggest gains showing up in long context workloads where the cost of recomputing versus reusing cache matters most. As of their most recent public updates, the system is running across thousands of nodes and processing over 100 billion tokens per day in production, not a lab benchmark.

One detail that is easy to miss: Mooncake does not assume every request should be processed no matter what. Under heavy load, it uses a prediction based early rejection policy, essentially deciding as early as possible that a request cannot realistically meet its latency target and rejecting it immediately, rather than letting it consume GPU resources partway through and then failing anyway. That is a meaningfully different design philosophy from most traditional web backends, where the instinct is usually to queue and retry rather than reject early.

The rest of the ecosystem is converging on the same idea

Mooncake is not an isolated experiment. By 2026, prefill/decode disaggregation has effectively become the industry default rather than a niche optimization.

vLLM, one of the most widely used open source inference engines, supports disaggregated serving natively, using NIXL as its standard transfer mechanism alongside NVIDIA Dynamo. SGLang offers a similar setup through its own router and disaggregation mode, and its RadixAttention design gives it a particular advantage for workloads with heavy prompt reuse. LMDeploy has added support through a comparable mechanism. NVIDIA's own Dynamo platform builds a routing and orchestration layer on top of this pattern for large scale deployments.

Perhaps the clearest sign that this has moved from research to infrastructure standard is llm-d, a project from Red Hat, IBM, and Google that brings Kubernetes native disaggregated inference on top of vLLM. It was donated to the CNCF in early 2026, and its default configuration reportedly delivers a meaningful performance improvement with no manual tuning required, with later releases showing significantly lower per token latency for large models on modern GPU hardware.

Production adoption backs this up. Reports throughout 2026 point to Meta, LinkedIn, Mistral, and Hugging Face running vLLM with disaggregated serving in production, and DeepSeek and Google's Gemini team have both discussed similar architectures at scale. Even at the hardware level, the idea is expanding. AWS partnered with Cerebras this year to bring disaggregated inference to Amazon Bedrock, using one type of chip for prompt processing and an entirely different type of chip for token generation, running side by side in the same cloud region.

Pushing disaggregation further: splitting more than just prefill and decode

Once you accept the idea that different parts of inference have different resource profiles, it is natural to ask whether the split should go deeper than just prefill versus decode. Some of the newest research does exactly that.

ByteDance's MegaScale Infer system takes disaggregation a step further by splitting the attention computation from the feedforward network (FFN) computation within the decode phase itself, since these two sub components also have different bandwidth and compute characteristics. Their reported deployment runs across close to 10,000 GPUs, using one GPU type optimized for the bandwidth heavy attention work and a different GPU type for the compute heavy expert layers.

There is also work extending this pattern to multimodal models, where an additional pool of instances handles encoding images, audio, or video before the text pipeline even begins, effectively creating a three stage pipeline of encode, prefill, and decode rather than just two.

The direction is clear: the industry is moving toward treating inference as a pipeline of specialized stages that can each be placed on the hardware best suited to them, rather than a single monolithic job.

Scheduling: the part nobody talks about enough

A disaggregated architecture is only as good as the scheduler deciding how to use it. This is where most of the ongoing research effort is actually concentrated, more so than the transfer mechanics themselves.

The key metric these systems optimize for is not raw throughput but goodput, meaning the number of requests that are fully completed within their latency service level objective. A request that gets 80 percent of the way through and then misses its deadline has still consumed GPU time and produced nothing useful, so it counts against the system rather than for it. Recent scheduling research also focuses on request imbalance, meaning what happens when a burst of long prompts arrives and threatens to starve the decode pool of attention, since prefill and decode instances scale independently and a mismatch between the two creates its own kind of bottleneck.

This is a genuinely different scheduling problem from most systems most engineers have built before. It is closer to job scheduling in a data center than request routing in a typical web service, because the "jobs" have two dependent phases running on different machines with a real time data transfer between them.

When disaggregation is not worth it

None of this is free. Splitting prefill and decode introduces real operational cost: two separate clusters to monitor, a KV cache transfer layer that can fail independently of either compute pool, cache miss handling, and a more complex deployment story overall. Teams that adopt this pattern before they actually need it often spend a meaningful amount of extra engineering time in the first several months for little practical benefit.

A reasonable way to think about whether disaggregation is worth adopting:

The signal worth paying attention to is a correlation between long prompt arrivals and latency spikes in ongoing token generation. If that pattern is not showing up in your metrics, the complexity of disaggregation is probably not paying for itself yet, and the better investment is tuning batch sizes and KV cache management on a single unified cluster.

Closing thoughts

Disaggregated prefill/decode serving is a good example of a broader trend in system design right now: the systems that matter most in 2026 are the ones built specifically for how large language models actually behave, rather than general purpose infrastructure patterns borrowed from earlier eras of web architecture. Prefill and decode are fundamentally different computational problems, and treating them that way, with dedicated hardware pools and a fast cache transfer layer between them, has produced real, measurable gains in production at some of the largest AI companies in the world.

It is also a useful case study in judgment. The right answer is not "always disaggregate." It is understanding your own workload well enough to know whether the added complexity buys you something real, which is the same question that sits underneath almost every good system design decision.

Why AI Assistants Have an Amnesia Problem. And How We're Fixing It

Sarthak Rawat — Sat, 06 Jun 2026 08:43:32 +0000

A deep dive into HGVM: the memory architecture that makes AI agents remember like humans, forget like humans, and learn like humans.

There is a moment every frequent AI user has experienced. You spent twenty minutes in a conversation explaining your project, your constraints, your preferences, your stack. The assistant was helpful, precise, contextually aware. Then you closed the tab.

The next day, you open a new conversation. The assistant greets you like a stranger.

You explain everything again.

This is not a minor inconvenience. It is a fundamental architectural failure, and it is not the only one. On the other side of the spectrum are systems that remember everything forever: every offhand comment, every temporary preference, every piece of stale context from six months ago. These systems don't forget, but they accumulate noise until their memory becomes a liability. They surface outdated information as confidently as current facts. They become sycophantic, anchored to what you said before rather than what is true now.

Neither model is how memory should work. Neither model is how human memory works.

We built HGVM, Hierarchical Graph-Vector Memory, to fix this. This blog explains what it is, how it works, and why we think it represents a meaningfully different approach to AI agent memory.

The Human Memory Analogy

Human memory is not a database. It does not store everything with equal fidelity and retrieve it on demand. It is a dynamic, adaptive system that continuously organizes, compresses, reinforces, and discards information based on relevance, recency, and repeated exposure.

When you learn something new, your brain does not simply file it. It connects it to existing structures. It consolidates related fragments into coherent abstractions during sleep. It strengthens memories that are repeatedly accessed and allows others to fade. And crucially, it reflects, it notices patterns across experiences and crystallizes them into durable knowledge that shapes future reasoning.

HGVM is built on this intuition. Memory should be a managed resource, not a growing pile. It should be structured hierarchically, retrieved selectively, compressed into abstractions over time, and actively pruned when it is no longer relevant. Four behaviors, working together.

The Four Pillars of HGVM

1. Hierarchical Structure

Most existing memory systems store memories as a flat collection of text chunks with embeddings. When you query, you run a vector similarity search across all of them and return the top results. This works adequately at small scale, but it fails in two important ways as the memory grows.

First, it causes context leakage, semantically similar but contextually irrelevant memories surface because the retrieval system has no way to distinguish between "similar in meaning" and "relevant to this situation." A question about your Python project might surface memories about a Python script you wrote for a completely different purpose two months ago.

Second, it is expensive. Vector search across all memories grows linearly. The more you remember, the slower and noisier retrieval becomes.

HGVM organizes memory into a strict four-layer hierarchy:

Domain → Category → Topic → Episode

A Domain is the broadest grouping: Work, Health, Personal, Finance. A Category is a sub-domain: ProjectX, Morning Runs, Family. A Topic is a coherent theme within a category: Backend API Design, Sprint Planning, Dad's Hospital Visit. An Episode is a single atomic memory item attached to a topic.

When a query arrives, the system routes at the Domain and Category level first, a cheap semantic match against summaries, and then runs vector search only within the relevant sub-graph. The candidate set is small. Retrieval is precise. Context leakage is dramatically reduced because irrelevant branches are never even considered.

This is the structural foundation everything else builds on.

2. Active Forgetting

Forgetting is not a failure mode. It is a feature.

The reason most AI memory systems remember everything forever is that forgetting feels dangerous, what if you delete something important? The result is a system that accumulates noise indefinitely, where the signal-to-noise ratio of the memory store degrades over time and retrieval quality degrades with it.

HGVM takes a different position: forgetting should be tiered by memory type, not uniform. Not all memories deserve the same treatment.

We define five memory classes, each with its own decay behavior:

Memory Class	What It Stores	Decay Rate
`permanent`	Explicit identity facts ("I am vegetarian", "never forget this")	Never decays
`semantic`	Consolidated summaries and learned patterns	Very slow
`preference`	Stable user preferences	Very slow
`observation`	Raw observations and task outcomes	Medium
`task_state`	Active work state and in-progress details	Fast

A permanent memory :- something the user explicitly told the system to remember, never decays. A preference :- "I prefer TypeScript", fades very slowly, remaining useful for months. A task_state memory :- "currently blocked on the auth middleware", fades quickly, becoming stale within days because task context changes fast.

The decay formula itself is borrowed from cognitive science. We use an exponential decay function where the strength of each memory degrades over time since last access:

new_strength = current_strength × exp(−λ × days_since_access)

Each memory class has its own λ value. When strength falls below a minimum threshold, the memory is soft-deleted :- marked as invalid but preserved in the graph for auditability. Nothing is ever hard-deleted during forgetting.

Retrieval is reinforced too: every time a memory is returned in a query, its strength is boosted. Memories that are repeatedly useful stay strong. Memories that are never relevant fade gracefully.

This is the forgetting curve most existing systems ignore. HGVM embraces it as a design principle.

3. Consolidation

A topic with fifty raw episode nodes is not useful. Fifty individually embedded fragments retrieved piecemeal create noisy, redundant context. At some point, accumulated raw memories need to be compressed into something denser and more useful.

This is consolidation.

HGVM runs a scheduled consolidation pipeline that works as follows: when a topic accumulates enough raw episodes, the system clusters them by semantic similarity, extracts atomic facts from each cluster, generates a concise summary from those facts, and then runs a batched verification pass to confirm the summary is faithful to the source material.

The verification step is non-negotiable. We check for three failure conditions: missing facts (something in the source that did not make it into the summary), altered facts (something that changed in meaning), and contradicted facts (something that directly conflicts with the source). If any of these conditions are non-empty, the consolidation is rejected and logged. No summary episode is created until verification passes completely.

When verification passes, the system creates a new semantic episode with subtype consolidation_summary, linked to the source episodes via SUMMARIZES relationships. Critically, the source episodes are never deleted. Consolidation adds compressed knowledge — it does not destroy provenance.

The retrieval system then naturally prefers the summary for most queries, it is denser, more representative, and semantically richer, while the raw episodes remain available for queries that need granular detail.

This mirrors how human memory consolidates during sleep: the raw experiences remain somewhere in the system, but what you access day-to-day is the compressed, organized version.

4. Reflection

Consolidation compresses. Reflection learns.

Reflection is the most novel component of HGVM. While consolidation creates summaries within a single topic, reflection operates across topics and sessions, looking for higher-order patterns in behavior, preference, and reasoning.

Imagine a user who across ten sessions has: rejected Java twice, chosen Python for three consecutive projects, consistently used FastAPI over Django, and complained about verbose enterprise frameworks. No single episode says "this user values developer productivity." But the pattern across episodes does.

Reflection detects this. It fetches recent valid episodes from the non-reflective memory classes :- permanent, preference, task_state, and observation, groups them into coherent evidence bundles, and generates candidate higher-order abstractions. These are persisted as semantic episodes with subtype reflection_pattern, linked to their supporting evidence.

The result: the next time the user asks for a project scaffold recommendation, the system does not just recall that they used FastAPI before. It retrieves the reflection, "this user prioritizes developer productivity over ecosystem conservatism", and reasons from that durable insight.

There is one hard constraint we enforce at the architecture level: reflection memories can never be used as evidence for future reflections. This prevents a feedback loop where the system reflects on its own abstractions, compounding distortions over time. Reflection operates only on primary evidence. This constraint is enforced at the database query level, not just the prompt level.

How HGVM Compares to What Exists

The agent memory landscape has become more sophisticated in the last two years. It is worth being honest about where existing systems land and where HGVM goes further.

MemGPT / Letta introduced the idea of paging between active context and external memory using function calls, and Letta added sleep-time consolidation agents. HGVM builds directly on these ideas but adds hierarchical routing, tiered forgetting, and the reflection pipeline.

Mem0 offers a clean multi-signal retrieval system (semantic + keyword + entity) with good API design. It does not offer hierarchical memory organization, tiered decay, or reflection.

Zep / Graphiti provide excellent graph-based memory with temporal reasoning and the bitemporal model (facts carry both a valid time and a transaction time). HGVM adopts the bitemporal approach for contradiction handling and uses Graphiti's valid_at / invalid_at pattern for soft deletion.

ChatGPT Memory is a static injection system, remembered facts are prepended to context. This causes cross-domain leakage (facts from unrelated contexts surface inappropriately) and sycophancy (the system defers to stored preferences even when they are stale or wrong). PersistBench has documented both failure modes.

HGVM is not a critique of any of these systems. Each solved a real problem. HGVM synthesizes the best ideas :- graph structure, temporal modeling, consolidation, multi-signal retrieval and adds forgetting as a first-class feature and reflection as a novel capability.

The Multi-Agent Layer

HGVM is not just a memory system. It is a memory system designed for an agent society.

Four specialized LLM agents share the same memory graph and collaborate on tasks:

Planner :- memory-first orchestration. Queries memory before planning anything. Owns task-state memory. Decides what gets persisted after each turn.
Executor :- task execution. Writes observation memories. Lower trust than Critic for factual claims.
Critic :- factual review and contradiction correction. Highest trust for permanent and preference memories. Can invalidate incorrect stored facts.
Reflection :- scheduled background agent. Generates semantic patterns from non-reflective evidence. Does not participate in the synchronous chat loop.

A critical design decision: no agent writes directly to the memory graph. All persistent memory writes go through a single MemoryManager service. This enforces a trust hierarchy; when Planner and Critic disagree about a stored fact, the Critic's version wins for permanent and preference memories. When write conflicts occur simultaneously, both versions are preserved with provenance tags until a resolution signal arrives.

This means the memory is not just accurate, it is accountable. Every episode knows which agent created it, when it was created, and whether it was ever superseded.

What the System Looks Like

The HGVM dashboard is built to make memory formation visible in real time. The Memory Graph page shows the full hierarchy as an interactive graph on a dark background: Domain hexagons in teal, Category rectangles, Topic circles with summary previews, and Episode nodes color-coded by strength, bright cyan for strong memories, yellow for medium, orange for fading.

New episodes appear with a fade-in animation. Strength color transitions animate as decay runs. When consolidation fires on a topic, a brief pulse effect signals the compression happening. Reflection outputs appear as double-ring nodes, visually distinct from raw and summary episodes.

A live Memory Activity Feed alongside the chat interface shows every memory operation in real time, green for ADD, blue for QUERY, red for INVALIDATE, purple for CONSOLIDATE, teal for REFLECT, grey for FORGET. You can watch the system's memory evolve turn by turn.

An Analytics page tracks memory growth by class over time, token savings from compression, consolidation verification health, reflection utilization rates, retrieval configuration, and topic reuse effectiveness.

The goal is to make memory legible, not just to researchers, but to anyone using the system.

The Hard Problems We Have Not Solved

We want to be clear about what remains genuinely difficult, because intellectual honesty matters more than clean marketing.

Decay parameter tuning. The λ values for each memory class are educated starting points informed by cognitive science and ablation experiments. The right values for a specific deployment domain — medical, coding, creative writing, likely differ. Universal optimal values do not exist yet.

Machine unlearning. GDPR's right to be forgotten is straightforward for the graph: a cascade deletion removes all a user's data atomically. It is not straightforward for the LLM itself. If an LLM has been exposed to a user's memories in-context during inference, traces of that exposure may persist in ways that are not cleanly deletable. Machine unlearning for in-context exposure is still an open research problem.

Semantic conflict resolution at scale. When two agents write contradictory facts at the same timestamp, the trust hierarchy handles it. But there are edge cases :- domain-dependent conflicts, partial contradictions, nuanced updates that partially overlap with stored facts — where the right resolution depends on context that no static rule captures. We use a conservative approach (preserve both, resolve later) but this is a heuristic, not a solution.

Reflection quality at low evidence density. When a user is new and has generated few episodes, reflection has little to work with. Thin-evidence reflections risk being generic or incorrect. We suppress reflections with insufficient supporting evidence, but the threshold is calibrated manually rather than learned.

These are real limitations. We name them because they are the honest frontier of this problem space, and because solving them is where the interesting research goes next.

What We Are Building Toward

HGVM v2.0 is being built for the Global AI Hackathon Series with Qwen Cloud with a tight deadline, but the ideas behind it are not hackathton ideas. The question of how AI agents should manage long-term memory is one of the most practically important open questions in applied AI right now.

Every production AI assistant deployment faces this problem. Every enterprise deploying agents across thousands of users faces this problem. Every personal assistant that fails to remember what you told it last week fails because of this problem.

We think the right direction involves all four pillars working together: hierarchical structure for precision, tiered forgetting for noise control, consolidation for compression, and reflection for higher-order learning. Not as research curiosities but as production-grade engineering.

The full architecture, schema, pipeline specifications, and implementation plan are documented internally and will be shared progressively as the system matures. The dashboard will be publicly demoed when it is stable.

If you are working on agent memory, long-term personalization, or multi-agent coordination, we would genuinely like to hear what you think. The hard problems above are not going to be solved alone.

HGVM is being built as part of a focused 30-day engineering sprint. Architecture is frozen. Implementation is active.

If this resonates with work you are doing, reach out.

Why AI Assistants Have an Amnesia Problem. And How We're Fixing It

Sarthak Rawat — Sat, 06 Jun 2026 08:30:00 +0000

A deep dive into HGVM: the memory architecture that makes AI agents remember like humans, forget like humans, and learn like humans.

The next day, you open a new conversation. The assistant greets you like a stranger.

You explain everything again.

Neither model is how memory should work. Neither model is how human memory works.

We built HGVM, Hierarchical Graph-Vector Memory, to fix this. This blog explains what it is, how it works, and why we think it represents a meaningfully different approach to AI agent memory.

The Human Memory Analogy

The Four Pillars of HGVM

1. Hierarchical Structure

Second, it is expensive. Vector search across all memories grows linearly. The more you remember, the slower and noisier retrieval becomes.

HGVM organizes memory into a strict four-layer hierarchy:

Domain → Category → Topic → Episode

This is the structural foundation everything else builds on.

2. Active Forgetting

Forgetting is not a failure mode. It is a feature.

HGVM takes a different position: forgetting should be tiered by memory type, not uniform. Not all memories deserve the same treatment.

We define five memory classes, each with its own decay behavior:

Memory Class	What It Stores	Decay Rate
`permanent`	Explicit identity facts ("I am vegetarian", "never forget this")	Never decays
`semantic`	Consolidated summaries and learned patterns	Very slow
`preference`	Stable user preferences	Very slow
`observation`	Raw observations and task outcomes	Medium
`task_state`	Active work state and in-progress details	Fast

The decay formula itself is borrowed from cognitive science. We use an exponential decay function where the strength of each memory degrades over time since last access:

new_strength = current_strength × exp(−λ × days_since_access)

Retrieval is reinforced too: every time a memory is returned in a query, its strength is boosted. Memories that are repeatedly useful stay strong. Memories that are never relevant fade gracefully.

This is the forgetting curve most existing systems ignore. HGVM embraces it as a design principle.

3. Consolidation

This is consolidation.

This mirrors how human memory consolidates during sleep: the raw experiences remain somewhere in the system, but what you access day-to-day is the compressed, organized version.

4. Reflection

Consolidation compresses. Reflection learns.

How HGVM Compares to What Exists

The agent memory landscape has become more sophisticated in the last two years. It is worth being honest about where existing systems land and where HGVM goes further.

Mem0 offers a clean multi-signal retrieval system (semantic + keyword + entity) with good API design. It does not offer hierarchical memory organization, tiered decay, or reflection.

The Multi-Agent Layer

HGVM is not just a memory system. It is a memory system designed for an agent society.

Four specialized LLM agents share the same memory graph and collaborate on tasks:

Planner :- memory-first orchestration. Queries memory before planning anything. Owns task-state memory. Decides what gets persisted after each turn.
Executor :- task execution. Writes observation memories. Lower trust than Critic for factual claims.
Critic :- factual review and contradiction correction. Highest trust for permanent and preference memories. Can invalidate incorrect stored facts.
Reflection :- scheduled background agent. Generates semantic patterns from non-reflective evidence. Does not participate in the synchronous chat loop.

This means the memory is not just accurate, it is accountable. Every episode knows which agent created it, when it was created, and whether it was ever superseded.

What the System Looks Like

The goal is to make memory legible, not just to researchers, but to anyone using the system.

The Hard Problems We Have Not Solved

We want to be clear about what remains genuinely difficult, because intellectual honesty matters more than clean marketing.

These are real limitations. We name them because they are the honest frontier of this problem space, and because solving them is where the interesting research goes next.

What We Are Building Toward

If you are working on agent memory, long-term personalization, or multi-agent coordination, we would genuinely like to hear what you think. The hard problems above are not going to be solved alone.

HGVM is being built as part of a focused 30-day engineering sprint. Architecture is frozen. Implementation is active.

If this resonates with work you are doing, reach out.

From Scripts to Systems: What Learning Kestra Taught Me About Workflow Orchestration

Sarthak Rawat — Fri, 08 May 2026 18:32:13 +0000

Introduction

Before starting the Kestra Fundamentals course, I used to think workflow orchestration was mostly about scheduling scripts.

Run a cron job.
Execute a Python script.
Send an email.
Done.

But the deeper I went into workflow orchestration, the more I realized modern systems are far more complex than that.

Today, applications depend on APIs, databases, cloud services, analytics pipelines, notifications, and event-driven systems all working together in the correct order.

The challenge is no longer just writing code.

The real challenge is coordinating systems reliably.

That’s where workflow orchestration comes in.

Through the Kestra Fundamentals course by WeMakeDevs, I learned how orchestration platforms help engineers build reliable, automated, observable workflows instead of disconnected scripts.

And honestly, this course changed the way I think about backend systems.

What Workflow Orchestration Actually Means

The easiest analogy for workflow orchestration is an orchestra.

Different musicians play different instruments.
Some enter early.
Some wait.
Some depend on others.

Without coordination, everything becomes noise.

The same thing happens in software systems.

You might have:

APIs fetching data
scripts processing it
databases storing it
analytics pipelines transforming it
notifications sending results

All these systems need to work together in the right order.

Workflow orchestration is the layer that coordinates all of this.

It helps with:

sequencing tasks
handling dependencies
retries and failures
automation
scheduling
monitoring
observability

Instead of writing isolated scripts and hoping they work, orchestration platforms let you design reliable systems.

Why Kestra Felt Different

One thing I immediately liked about Kestra was how structured everything felt.

Workflows are defined declaratively using YAML.

Instead of manually stitching together scripts, you define workflows in a clean and readable format.

Kestra also provides:

execution tracking
visual workflow monitoring
retries
scheduling
triggers
plugins
reusable blueprints

What stood out to me most was that Kestra didn’t feel like just another automation tool.

It felt like a system orchestration platform.

Core Concepts That Finally Clicked

Flows

A Flow is the main orchestration unit in Kestra.

This is where the overall workflow is defined:

tasks
triggers
inputs
outputs
orchestration logic

At first, I thought flows were just pipelines.

But eventually, they started feeling more like system blueprints.

Tasks

Tasks are the individual units of work.

For example:

calling an API
running a script
querying a database
sending notifications

One thing I liked about Kestra’s design is how composable tasks feel.

Each task focuses on one responsibility.

That makes workflows much easier to reason about.

Inputs and Outputs

This was one of the concepts that made orchestration really click for me.

Inputs allow workflows to receive data dynamically.
Outputs allow tasks to pass data to other tasks.

So instead of disconnected scripts, you get connected systems.

Example:

Task A fetches API data → Task B processes it → Task C stores it.

The clean data flow model makes workflows much easier to scale and debug.

Triggers

Triggers are what make workflows truly automated.

Instead of manually executing workflows, Kestra can trigger them based on:

schedules (cron)
API events
file arrivals
workflow completions

This shifts workflows from manual execution to event-driven systems.

Expressions

Expressions were another powerful concept.

Kestra uses templating syntax to dynamically reference values and outputs.

Example:

{{ outputs.task_id.value }}

This allows workflows to become dynamic and programmable instead of static pipelines.

Flowable Tasks Changed How I Think About Orchestration

This section was probably one of the biggest mindset shifts for me.

Flowable tasks are not just tasks that execute work.

They control orchestration logic itself.

This includes:

loops
conditionals
parallel execution
branching
subflows

Subflows especially felt similar to reusable functions in programming.

Define once.
Reuse everywhere.

That realization made orchestration feel much closer to software architecture than simple automation.

Execution, Observability, and Reliability

One thing many beginners underestimate is observability.

Running workflows is only part of the problem.

Understanding:

what failed
why it failed
where it failed

is equally important.

Kestra’s execution tracking and logs make workflows much easier to debug.

Instead of guessing what happened, you can inspect executions step-by-step.

Retries and execution monitoring also make workflows significantly more reliable.

Secrets, Plugins, and Blueprints

Another thing I appreciated was how practical the platform felt.

Secrets

Secrets allow sensitive values like:

API keys
credentials
tokens

to stay secure instead of being hardcoded.

Plugins

Kestra has a huge plugin ecosystem.

This means workflows can integrate with:

APIs
cloud services
databases
scripting environments
data systems

without constantly reinventing integrations.

Blueprints

Blueprints are reusable workflow templates.

This is extremely useful when:

learning orchestration
prototyping workflows quickly
exploring integrations

Instead of starting from scratch, you can learn from working examples.

The Project I Built

One of the biggest things I wanted after completing the course was to actually apply what I learned instead of stopping at theory.

So I built a small workflow project using Kestra, an automated daily sales report pipeline.

The goal of the workflow was simple:

Automatically fetch sales data, process it, generate a report, and send it through email without any manual intervention.

What the workflow does

The workflow follows this sequence:

A cron trigger starts the workflow automatically on schedule.
Sales data is fetched from a PostgreSQL database.
The raw data is processed and transformed using Python scripts.
A summary report is generated.
The report is exported as a CSV file.
The final report is sent via email.

This project helped me understand how orchestration systems coordinate multiple dependent tasks reliably.

What I Learned While Building It

Building this workflow made several orchestration concepts finally click for me.

Dependencies and execution order

Each task depended on outputs from previous tasks.

For example:

report generation depends on transformed data
email sending depends on successful CSV generation

This made workflow dependencies feel much more real than isolated scripts.

Inputs and Outputs

Passing data between tasks using outputs was one of the most interesting parts.

Instead of manually handling intermediate files everywhere, the workflow itself became the coordination layer.

Automation Design

The trigger system showed how workflows can become completely automated.

Once configured, the entire pipeline could run on schedule without manual execution.

Observability and Reliability

Execution logs and task tracking made debugging much easier.

Instead of guessing where failures happened, I could inspect each task execution step-by-step.

That visibility is something normal scripts usually lack.

System Thinking

The biggest realization for me was this:

I wasn’t just writing scripts anymore.

I was designing a system where:

tasks coordinate together
data flows between steps
failures can be monitored
workflows execute automatically

That shift in mindset was probably the most valuable part of the entire course.

Final Thoughts

The biggest shift for me after learning Kestra was realizing this:

Modern engineering isn’t just about writing code.

It’s about designing systems that:

coordinate work reliably
automate execution
handle dependencies
recover from failures
provide visibility into what’s happening

Workflow orchestration sits at the center of all of this.

And Kestra made these ideas surprisingly approachable.

I originally started this course thinking orchestration was just advanced scheduling.

I finished it understanding why orchestration is such an important part of modern software systems.

Stop Building Reactive AI — Build AI That Acts Before You Ask

Sarthak Rawat — Sat, 25 Apr 2026 20:41:22 +0000

This is a submission for the OpenClaw Writing Challenge

The Problem With Most AI

Most AI systems today follow the same pattern:

You ask → it responds

Chatbots, assistants, copilots — they all wait.

But real-world problems don't work like that.

You don't notice when a task is slowly being ignored
You don't realize work is at risk until it's too late
You don't ask for help because you don't know there's a problem yet

AI that waits for prompts is fundamentally limited.

It reacts to awareness — but awareness is often the problem.

Reactive AI waits for input before acting

Enter OpenClaw

When I first heard about OpenClaw, I thought of it as just another AI assistant — something that sits on your machine and answers questions faster.

I was wrong.

OpenClaw describes itself as a personal AI assistant that runs on your own devices and answers you on the channels you already use — Telegram, Slack, Discord, WhatsApp, and dozens more. But the real insight isn't in the channels. It's in the Gateway.

The Gateway is OpenClaw's local control plane: a persistent process that's always running, always watching, and always ready to act. That changes everything.

Because now the question isn't just "what can my AI respond to?"

The question becomes: "what should my AI notice?"

The Shift: From Reactive to Proactive AI

After building a workflow monitoring agent on top of OpenClaw, I realized something:

The real value of AI isn't answering questions.
It's noticing problems before you do.

Instead of the classic model:

User → Prompt → AI Response

OpenClaw lets you build something fundamentally different:

Observe → Detect → Decide → Intervene

The AI is no longer just a tool you use — it becomes something that works alongside you, even when you're not thinking about it.

Proactive AI continuously observes, detects, and intervenes

How OpenClaw Makes This Possible

The Gateway: Always On, Never Waiting

Most AI tools live in a tab you have to open. OpenClaw's Gateway runs as a system service (via launchd on macOS, systemd on Linux). It's always on — like a background process for your life.

This is the prerequisite for proactive AI. You can't observe the world if you only wake up when someone talks to you.

Skills: Teaching Your Agent What to Watch

OpenClaw's extensible Skills system is what lets you encode custom logic into your agent. A skill is just a folder with a SKILL.md file — plain, hackable, yours.

For my workflow monitoring agent, I wrote a skill that:

Checks open tasks across my project board
Measures how many days have passed since the last commit
Counts files that are modified but uncommitted

None of this required complex infrastructure. Just a skill, some shell commands, and OpenClaw's tool-calling loop.

Cron-Style Loops: The Heartbeat of Proactive AI

OpenClaw supports scheduled and recurring agent runs. This is the heartbeat of the proactive pattern:

check → analyze → act → repeat

Every 20 minutes, my agent wakes up, runs the skill, evaluates current state, and decides whether to intervene. Most of the time, it does nothing. But when it detects something worth flagging — a task ignored for 3 days, an uncommitted spike of files — it reaches out.

Simple loop: check → analyze → act → repeat

Channels: Reaching You Where You Already Are

Because OpenClaw routes through messaging platforms, the intervention doesn't feel like a notification from yet another app.

It feels like a message from someone paying attention.

When my agent pinged me on Telegram — "This task has been sitting for 3 days. You'll likely miss the deadline — prioritize it today." — I actually acted on it. Because it arrived in the same place my real messages do, in plain language, at the right moment.

That's the difference between logging a risk and actually changing behavior.

What Changes When AI Stops Waiting

1. It Works Without Being Asked

You don't need to remember to use it.

It's already watching:

patterns
delays
risks

The system becomes ambient — always present, never intrusive.

2. It Solves Problems You Didn't See

Reactive AI can only solve known problems.

Proactive AI surfaces unknown problems:

forgotten tasks
risky workflows
accumulating technical debt

This is where most real-world value comes from.

3. It Feels Less Like a Tool, More Like a System

A chatbot feels like software.

A proactive OpenClaw agent feels like:

a safety net
a second layer of awareness
something that actively supports your workflow

That shift is subtle — but powerful.

What I Learned Building This

Simplicity Beats Complexity

The system didn't need to be perfect.

Even simple signals — like "days since last commit" — were enough to detect meaningful risk. OpenClaw's skill system made it easy to start small and iterate.

Context Matters More Than Rules

Hardcoded thresholds failed quickly.

What mattered was context. Five uncommitted files might be fine — or catastrophic, depending on the deadline. OpenClaw's LLM-powered evaluation layer is what made the difference: instead of a rule engine, I had something that could interpret, not just calculate.

Small Interventions Create Big Impact

The agent didn't need to automate everything.

It just needed to say the right thing at the right time, through the right channel.

That was enough to change behavior.

How to Build This Yourself with OpenClaw

If you want to try this pattern, here's where to start:

1. Get OpenClaw Running

curl -fsSL https://openclaw.ai/install.sh | bash
openclaw onboard --install-daemon

The onboarding wizard walks you through model provider setup, API keys, and Gateway configuration in about 2 minutes.

2. Connect a Channel

Telegram is the easiest first channel. Once it's connected, your agent can reach you wherever you are — no new app needed.

3. Write a Simple Skill

Create a folder at ~/.openclaw/workspace/skills/workflow-monitor/ with a SKILL.md file. Define what your agent should observe: stale tasks, missed commits, overdue tickets — whatever your workflow surfaces.

4. Set Up a Loop

Schedule your skill to run every 15–30 minutes. This is your agent's heartbeat. Most runs will be quiet. The ones that aren't are the ones that matter.

5. Generate Interventions, Not Just Data

The difference between useful and useless proactive AI is this:

Don't just log:

"Task is overdue"

Instead, have your agent reason about it:

"This task has been ignored for 3 days. You'll likely miss the deadline — prioritize it today."

OpenClaw's LLM layer handles this naturally. You don't need to write the reasoning — you just need to give it the signal.

Where This Goes Next

This pattern isn't limited to developer workflows.

Anywhere you have data + delay + risk, you can apply proactive AI with OpenClaw:

Sales pipelines → catching stalled deals before they die
Support systems → flagging SLA risks before they breach
Personal productivity → identifying burnout patterns before they compound

OpenClaw's open ecosystem — with ClawHub, community skills, and multi-platform channels — means you're not building from scratch. You're building on top of something already watching.

Final Thought

We've spent years improving how AI responds.

But maybe that's the wrong direction.

The next step isn't better answers.
It's AI that knows when to act.

OpenClaw gave me the infrastructure to stop asking "what can my AI respond to?" and start asking "what should my AI notice?"

That question changes everything.

Let's build a Production-Grade Bloom Filter in Python

Sarthak Rawat — Sat, 21 Mar 2026 19:04:40 +0000

Ever wondered how databases can tell you "this username is definitely not taken" in milliseconds without scanning millions of records? Or how caching systems avoid expensive database lookups for keys that don't exist? The secret is a probabilistic data structure called a Bloom Filter.

Let's build one from scratch :- with production features like persistence, serialization, and monitoring.

What's a Bloom Filter?

A Bloom filter is a space-efficient probabilistic data structure that tells you:

"Definitely not in the set" (100% certain)
"Probably in the set" (with a configurable false positive rate)

It's like a bouncer who sometimes lets the wrong person in but never turns away someone who should be there.

The Trade-off

Aspect	Traditional Set	Bloom Filter
Space	O(n) per element	~2-10 bytes per element
Time	O(1) average	O(k) where k ~ 5-10
False Positives	None	Configurable (0.1% - 5%)
Deletions	Supported	Not supported

For 10 million items, a hash set might use 500MB+ of memory. A Bloom filter with 1% false positive rate? Just ~12MB.

How It Works

Imagine a massive array of bits, all initially 0. When you add an element:

Run it through k different hash functions
Set the bits at positions h1(item), h2(item), ..., hk(item) to 1

To check if an item exists:

Run it through the same k hash functions
If ANY of those bits are 0 → definitely not in set
If ALL bits are 1 → probably in set

Adding "apple":
Hash1("apple") = 42 → set bit 42
Hash2("apple") = 157 → set bit 157
Hash3("apple") = 891 → set bit 891

Checking "banana":
Hash1("banana") = 42 → bit 42 is 1
Hash2("banana") = 203 → bit 203 is 0 → definitely not present!

The Math Behind It

Given:

n = expected number of elements
p = desired false positive rate

We can calculate:

Optimal bits: m = -n * ln(p) / (ln(2))²
Optimal hash functions: k = (m/n) * ln(2)

For 10,000 elements at 1% false positives:

m = 95,851 bits (~12KB)
k = 7 hash functions

Each query checks 7 bits. No matter if you have 10 or 10 million items.

Let's Build It

Storage Backend

First, we need a place to store bits. Let's make it extensible:

from abc import ABC, abstractmethod
import threading

class StorageBackend(ABC):
    @abstractmethod
    def get_bit(self, index: int) -> bool: pass

    @abstractmethod
    def set_bit(self, index: int) -> None: pass

    @abstractmethod
    def clear(self) -> None: pass

    @abstractmethod
    def to_bytes(self) -> bytes: pass

class InMemoryStorage(StorageBackend):
    def __init__(self, size: int):
        self.size_bits = size
        self.size_bytes = (size + 7) // 8
        self.bit_array = bytearray(self.size_bytes)
        self.lock = threading.RLock()

    def get_bit(self, index: int) -> bool:
        with self.lock:
            byte_index = index >> 3
            bit_index = index & 7
            return (self.bit_array[byte_index] >> bit_index) & 1 == 1

    def set_bit(self, index: int) -> None:
        with self.lock:
            byte_index = index >> 3
            bit_index = index & 7
            self.bit_array[byte_index] |= (1 << bit_index)

    def to_bytes(self) -> bytes:
        with self.lock:
            return bytes(self.bit_array)

Text Analysis

We need to normalize items before adding them:

from typing import List, Set
import mmh3

class HashFunction(Enum):
    MURMUR3 = "murmur3"
    SHA1 = "sha1"

class BloomFilter:
    def __init__(self, capacity: int, false_positive_rate: float = 0.01):
        self.capacity = capacity
        self.fpr = false_positive_rate
        self.size = self._calculate_size(capacity, false_positive_rate)
        self.num_hashes = self._calculate_num_hashes(self.size, capacity)
        self.storage = InMemoryStorage(self.size)

    def _calculate_size(self, n: int, p: float) -> int:
        m = - (n * math.log(p)) / (math.log(2) ** 2)
        return int(math.ceil(m))

    def _calculate_num_hashes(self, m: int, n: int) -> int:
        k = (m / n) * math.log(2)
        return max(1, int(math.ceil(k)))

    def _get_hash_values(self, item: str) -> List[int]:
        """Generate k hash positions using double hashing"""
        item_bytes = item.encode('utf-8')

        # Get two independent hash values
        hash1 = mmh3.hash64(item_bytes, seed=0, signed=False)[0]
        hash2 = mmh3.hash64(item_bytes, seed=42, signed=False)[0]

        positions = []
        for i in range(self.num_hashes):
            # Combine using double hashing formula
            pos = (hash1 + i * hash2) % self.size
            positions.append(pos)

        return positions

    def add(self, item: str) -> None:
        """Add an item to the filter"""
        for pos in self._get_hash_values(item):
            self.storage.set_bit(pos)

    def check(self, item: str) -> bool:
        """Check if item might be in the filter"""
        for pos in self._get_hash_values(item):
            if not self.storage.get_bit(pos):
                return False  # Definitely not present
        return True  # Probably present

Adding Statistics

Production systems need monitoring:

@dataclass
class BloomFilterStats:
    total_insertions: int = 0
    total_queries: int = 0
    false_positives: int = 0
    memory_usage_bytes: int = 0
    last_reset_time: datetime = None

    def get_false_positive_rate(self) -> float:
        if self.total_queries == 0:
            return 0.0
        return self.false_positives / self.total_queries

class BloomFilter:
    # ... previous code ...

    def __init__(self, capacity: int, fpr: float = 0.01, enable_stats: bool = True):
        # ... existing init ...
        self.stats = BloomFilterStats() if enable_stats else None

    def add(self, item: str) -> None:
        for pos in self._get_hash_values(item):
            self.storage.set_bit(pos)

        if self.stats:
            self.stats.total_insertions += 1
            self.stats.memory_usage_bytes = len(self.storage.to_bytes())

    def check(self, item: str) -> bool:
        result = True
        for pos in self._get_hash_values(item):
            if not self.storage.get_bit(pos):
                result = False
                break

        if self.stats:
            self.stats.total_queries += 1
            # Track false positives (would need external verification)

        return result

    def get_stats(self) -> Optional[dict]:
        if self.stats:
            return self.stats.to_dict()
        return None

    def estimate_current_capacity(self) -> float:
        """Estimate how many items are currently stored"""
        bits_set = sum(bin(b).count('1') for b in self.storage.to_bytes())
        ratio = bits_set / self.size

        if ratio >= 1.0:
            return float(self.capacity)

        # n = -m * ln(1 - ratio) / k
        n = -self.size * math.log(1 - ratio) / self.num_hashes
        return n

File Persistence

Make it survive restarts:

class FileStorage(StorageBackend):
    def __init__(self, size: int, filepath: str):
        self.size_bits = size
        self.size_bytes = (size + 7) // 8
        self.filepath = filepath
        self.lock = threading.RLock()
        self._load_or_create()

    def _load_or_create(self):
        if os.path.exists(self.filepath):
            with open(self.filepath, 'rb') as f:
                self.bit_array = bytearray(f.read())
        else:
            self.bit_array = bytearray(self.size_bytes)
            self._persist()

    def _persist(self):
        with open(self.filepath, 'wb') as f:
            f.write(self.bit_array)

    def set_bit(self, index: int) -> None:
        with self.lock:
            byte_index = index >> 3
            bit_index = index & 7
            self.bit_array[byte_index] |= (1 << bit_index)
            self._persist()  # Save after each change

# Usage
bf = BloomFilter(
    capacity=1_000_000,
    false_positive_rate=0.01,
    storage_backend="file",
    storage_path="bloom_filter.bin"
)

Serialization

Share filters across services:

class SerializationFormat(Enum):
    JSON = "json"
    PICKLE = "pickle"
    BASE64 = "base64"

class BloomFilter:
    # ... existing code ...

    def serialize(self, format: SerializationFormat = SerializationFormat.JSON) -> str:
        data = {
            'capacity': self.capacity,
            'false_positive_rate': self.fpr,
            'size': self.size,
            'num_hashes': self.num_hashes,
            'bit_array': base64.b64encode(self.storage.to_bytes()).decode('ascii')
        }

        if format == SerializationFormat.JSON:
            return json.dumps(data)
        elif format == SerializationFormat.PICKLE:
            return base64.b64encode(pickle.dumps(data)).decode('ascii')

    @classmethod
    def deserialize(cls, serialized: str, format: SerializationFormat = SerializationFormat.JSON) -> 'BloomFilter':
        if format == SerializationFormat.JSON:
            data = json.loads(serialized)
        else:
            data = pickle.loads(base64.b64decode(serialized))

        bf = cls(data['capacity'], data['false_positive_rate'])
        bf.storage.from_bytes(base64.b64decode(data['bit_array']))
        return bf

Set Operations

Merge multiple filters:

def merge(self, other: 'BloomFilter') -> None:
    """Merge another filter into this one (OR operation)"""
    if self.size != other.size or self.num_hashes != other.num_hashes:
        raise ValueError("Filters must have same parameters")

    self_bytes = self.storage.to_bytes()
    other_bytes = other.storage.to_bytes()

    merged = bytearray(a | b for a, b in zip(self_bytes, other_bytes))
    self.storage.from_bytes(bytes(merged))

def intersection(self, other: 'BloomFilter') -> 'BloomFilter':
    """Create new filter from intersection (AND operation)"""
    if self.size != other.size or self.num_hashes != other.num_hashes:
        raise ValueError("Filters must have same parameters")

    new_bf = BloomFilter(self.capacity, self.fpr)

    self_bytes = self.storage.to_bytes()
    other_bytes = other.storage.to_bytes()

    intersected = bytearray(a & b for a, b in zip(self_bytes, other_bytes))
    new_bf.storage.from_bytes(bytes(intersected))

    return new_bf

Complete Working Example

Here's the full code in action:

# Example 1: Basic Usage
bf = BloomFilter(capacity=1000, false_positive_rate=0.01)

# Add some items
items = ["apple", "banana", "cherry", "date"]
for item in items:
    bf.add(item)

# Test membership
print("apple in filter?", bf.check("apple"))      # True
print("grape in filter?", bf.check("grape"))      # False

# Example 2: With Statistics
bf = BloomFilter(capacity=10000, false_positive_rate=0.01, enable_stats=True)

for i in range(5000):
    bf.add(f"user_{i}")

stats = bf.get_stats()
print(f"Insertions: {stats['total_insertions']}")
print(f"Memory: {stats['memory_usage_bytes'] / 1024:.2f} KB")
print(f"Estimated capacity: {stats['estimated_capacity']:.0f}")

# Example 3: File Persistence
bf = BloomFilter(
    capacity=1_000_000,
    false_positive_rate=0.01,
    storage_backend="file",
    storage_path="my_filter.bin"
)

bf.add("persistent_data")
# Filter survives restarts!

# Example 4: Serialization
serialized = bf.serialize(SerializationFormat.JSON)
restored = BloomFilter.deserialize(serialized, SerializationFormat.JSON)

Gallery

Performance Numbers

Running on a modest laptop with 10 million items:

Operation	Time	Space
Insert 1 item	~0.5 µs	-
Check 1 item	~0.4 µs	-
Build 10M items	8 seconds	12 MB
Memory per item	-	1.2 bytes

The false positive rate? Exactly 1% as configured.

Real-World Applications

1. Cache Filtering

# Avoid cache lookups for keys that don't exist
cache_filter = BloomFilter(capacity=1_000_000, fpr=0.01)

def get_user(user_id):
    if not cache_filter.check(user_id):
        return None  # Definitely not in cache

    # Might be in cache - do the expensive lookup
    return cache.get(user_id)

2. Database Query Optimization

# Check if username exists without hitting the DB
username_filter = BloomFilter(capacity=100_000, fpr=0.001)

def username_available(username):
    if username_filter.check(username):
        # Probably taken - check database
        return not db.user_exists(username)
    return True  # Definitely available

3. Web Crawler Deduplication

# Remember visited URLs without storing them all
visited = BloomFilter(capacity=1_000_000_000, fpr=0.001)

def should_crawl(url):
    if visited.check(url):
        return False  # Probably visited
    visited.add(url)
    return True

4. Distributed Systems

# Sync filters across services
class DistributedBloomFilter:
    def __init__(self, capacity, fpr, redis_client):
        self.redis = redis_client
        self.key = "bloom_filter"
        self.filter = BloomFilter(capacity, fpr)

    def add(self, item):
        self.filter.add(item)
        # Periodically sync to Redis
        self.redis.set(self.key, self.filter.serialize())

    def check(self, item):
        # Try local first
        if self.filter.check(item):
            return True
        # Fall back to Redis if needed
        return self._check_redis(item)

Limitations and Gotchas

No Deletions

Bloom filters don't support deletion. Removing an item would require clearing bits that might belong to other items. For deletion support, check out Counting Bloom Filters (store counters instead of bits).

False Positives Increase with Saturation

As you add more items than capacity, the false positive rate increases:

Capacity: 1000 items
Items Added | Actual FPR
1000        | 0.99%
2000        | 3.2%
5000        | 15.8%

Hash Function Quality Matters

Use high-quality, independent hash functions. Our double hashing with Murmur3 works well, but avoid Python's built-in hash() it's not designed for this.

Where to Go From Here

This is just the beginning. Real production systems add:

Counting Bloom Filters: Support deletions with counters
Scalable Bloom Filters: Grow dynamically as you add items
Cuckoo Filters: Support deletions with better space efficiency
Partitioned Filters: Split across multiple machines
Bit-sliced Bloom Filters: SIMD-optimized for speed

The inverted index may be the heart of search engines, but the Bloom filter is the secret sauce that makes them fast. It's the reason databases can say "no" in microseconds, and caching systems can avoid millions of useless lookups.

Building TaskPilot: An AI Agent That Sees Your Screen and Takes Control

Sarthak Rawat — Mon, 16 Mar 2026 21:07:44 +0000

I created this blog for detailing about my project in the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

The Problem: Automation That Breaks the Moment the UI Changes

Every developer has been there. You write a Selenium script, it works perfectly, and then the website updates its CSS class names and the whole thing falls apart. You set up an RPA workflow, it runs fine for a week, and then someone moves a button and it starts clicking the wrong thing.

Traditional automation is brittle because it's blind. It relies on DOM selectors, API hooks, and hardcoded coordinates. It doesn't actually see the screen. It just pokes at it.

But humans don't automate that way. When you ask a colleague to "find the cheapest flight to New York and book it," they open a browser, look at the screen, read what's there, and make decisions based on what they see. They don't need an API. They don't need a DOM inspector. They just need eyes.

That's the gap TaskPilot fills. It's an AI agent that observes your screen the way a human would, understands what it sees using Gemini's multimodal vision, and executes actions based on natural language intent. No selectors. No APIs. No brittle scripts. Just vision, reasoning, and action.

Built for the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

The Architecture: Two Agents, One Interface

TaskPilot has two distinct execution environments that share a single Electron frontend:

Desktop Mode — A TypeScript/Node.js agent (clawd-cursor) that runs locally and controls your actual OS. It can open apps, type text, click buttons, switch windows, and execute multi-app workflows across your entire desktop.

Browser Mode — A Python WebSocket server (computer-use-preview) deployed on Google Cloud Run that spins up a Playwright browser, runs a Gemini Computer Use vision loop, and streams screenshots and reasoning back to the frontend in real time.

The Electron frontend connects to whichever mode the user selects. Desktop mode talks to a local REST API on 127.0.0.1:3847. Browser mode connects to a Cloud Run WebSocket endpoint. The UI is identical either way — live screenshots, a reasoning panel, an action timeline, and a voice input button.

The 5-Layer Pipeline: Why We Don't Always Need Vision

The most important architectural decision in TaskPilot is the layered execution pipeline in clawd-cursor. The core insight: taking a screenshot and sending it to a vision LLM is expensive and slow. Most tasks don't need it.

So we built five layers, each cheaper and faster than the last. The agent tries them in order and only escalates when the current layer can't handle the task.

Layer 0 — Browser (Playwright CDP)
For any task that involves a browser, we go straight to Chrome DevTools Protocol. No screenshots. No LLM. We read the DOM directly, find elements, and interact with them programmatically. This handles a huge chunk of web tasks instantly and for free.

Layer 1 — Action Router (Regex + Keyboard Shortcuts)
A pattern-matching layer that recognizes common intents and maps them to direct actions. "Scroll down" becomes a keyboard shortcut. "Copy" becomes Ctrl+C. "Open Notepad" becomes a shell command. No LLM involved. This layer handles the majority of simple desktop tasks in under a second.

// From action-router.ts — pattern matching before any LLM call
const SHORTCUT_PATTERNS = [
  { pattern: /scroll\s+down/i, action: () => keyboard.type(Key.PageDown) },
  { pattern: /copy/i,          action: () => keyboard.pressKey(Key.LeftControl, Key.C) },
  { pattern: /open\s+(\w+)/i,  action: (match) => shell.exec(`start ${match[1]}`) },
];

Layer 1.5 — Smart Interaction (1 LLM Call)
When pattern matching isn't enough, we make a single cheap text LLM call to plan the steps, then execute them via CDP or the accessibility tree. One call, no screenshots.

Layer 2 — A11y Reasoner (Accessibility Tree + Cheap LLM)
We read the OS accessibility tree — the structured representation of every UI element on screen — and feed it to a cheap text model. The model reasons about which element to interact with based on its label, role, and position. Still no screenshots.

Layer 3 — Computer Use (Vision LLM)
Only when all else fails do we take a screenshot and send it to Gemini or Anthropic Computer Use. This is the most powerful layer but also the most expensive. By the time we reach it, we've already handled 80%+ of tasks in the layers above.

The performance difference is dramatic:

Task	Without Pipeline	With Pipeline
Calculator (255×38)	43s (18 LLM calls)	2.6s (0 LLM calls)
Notepad (type hello)	73s	2.0s
File Explorer	53s	1.9s
Gmail compose	162s	21.7s (1 LLM call)

The Browser Agent: Gemini Computer Use in a Loop

The Python computer-use-preview service is where Gemini's multimodal capabilities really shine. It runs a tight vision loop:

Capture a screenshot of the Playwright browser
Send it to Gemini along with the task and conversation history
Gemini returns a function call (click, type, navigate, scroll, etc.)
Execute the action in Playwright
Capture the next screenshot and repeat

# From agent.py — the core Gemini Computer Use loop
async def agent_loop(self) -> str:
    while True:
        response = await self._client.aio.models.generate_content(
            model=self._model_name,
            contents=self._contents,
            config=GenerateContentConfig(
                tools=self._tools,
                system_instruction=SYSTEM_PROMPT,
            ),
        )

        # Extract function calls from response
        for part in response.candidates[0].content.parts:
            if part.function_call:
                result = await self._execute_action(part.function_call)
                self._contents.append(...)  # append result to history

        if response.candidates[0].finish_reason == FinishReason.STOP:
            return self.final_reasoning

Every screenshot is streamed back to the Electron frontend over WebSocket as a base64-encoded JPEG. The user watches the agent think and act in real time — they see the same screen the agent sees, with a cursor indicator showing exactly where it's about to click.

We also implemented screenshot pruning: only the last 3 screenshots are kept in the Gemini context window. Older ones are replaced with text summaries. This keeps token costs manageable for long-running tasks without losing context.

The WebSocket Server: Bridging Frontend to Agent

The Python server (server.py) is the glue between the Electron frontend and the browser agent. It manages session lifecycle, handles both browser and desktop modes, and routes voice input.

Each connection gets an AgentSession with a dedicated worker thread. Commands from the frontend go into a queue; the worker processes them sequentially. This keeps the WebSocket handler non-blocking while the agent does its work.

class AgentSession:
    def __init__(self, ws, loop):
        self._ws = ws
        self._loop = loop
        self._cmd_queue = queue.Queue()
        self._worker_thread = threading.Thread(
            target=self._worker_loop, daemon=True
        )
        self._worker_thread.start()

    def _worker_loop(self):
        while not self._closed:
            cmd = self._cmd_queue.get(timeout=0.5)
            if cmd["action"] == "run_agent":
                self._run_agent(cmd["query"], cmd["model"], cmd["mode"])

For desktop mode, the server proxies the task to the local clawd-cursor REST API via ClawdBridge. The frontend doesn't need to know which backend is handling the task — it just sends a message and receives a stream of screenshots and reasoning updates.

Voice Input: Talking to Your Agent

One of the more satisfying features to build was voice input. The user clicks the microphone button in the Electron UI, speaks their task, and the audio is sent to the Python server for transcription via Google Cloud Speech-to-Text.

# From voice_input.py
class VoiceTranscriber:
    def transcribe(self, audio_bytes: bytes) -> str:
        client = speech.SpeechClient()
        audio = speech.RecognitionAudio(content=audio_bytes)
        config = speech.RecognitionConfig(
            encoding=speech.RecognitionConfig.AudioEncoding.WEBM_OPUS,
            sample_rate_hertz=48000,
            language_code="en-US",
        )
        response = client.recognize(config=config, audio=audio)
        return response.results[0].alternatives[0].transcript

The transcribed text drops straight into the task input field. It's a small thing but it makes the agent feel much more natural to use — especially for longer, more complex instructions where typing is tedious.

Safety: The Agent Needs to Know When to Ask

Giving an AI agent control of your desktop is powerful. It's also potentially dangerous. We built a safety layer that classifies every action before executing it.

Actions fall into three tiers:

Auto — Navigation, scrolling, reading. Execute immediately.
Preview — Typing text, opening files. Show the user what's about to happen.
Confirm — Sending emails, deleting files, form submissions. Pause and require explicit approval.

The web dashboard (served by the Express server in clawd-cursor) shows pending confirmations in real time. The user can approve or reject any action before it executes. There's also a kill switch that immediately halts the agent.

// From safety.ts — tier classification
export class SafetyLayer {
  classify(action: InputAction): SafetyTier {
    if (BLOCKED_PATTERNS.some(p => p.test(action.description))) {
      return SafetyTier.Block;
    }
    if (CONFIRM_PATTERNS.some(p => p.test(action.description))) {
      return SafetyTier.Confirm;
    }
    if (PREVIEW_PATTERNS.some(p => p.test(action.description))) {
      return SafetyTier.Preview;
    }
    return SafetyTier.Auto;
  }
}

Provider-Agnostic by Design

We didn't want to lock TaskPilot into a single AI provider. The doctor command runs an interactive setup wizard that scans your environment, detects available providers, tests each one, and recommends the optimal pipeline configuration.

The provider is auto-detected from the API key format:

sk-ant-* → Anthropic
sk-* → OpenAI
AIza* → Gemini
Local endpoint → Ollama

You can also mix providers: use Ollama for cheap text tasks (free, runs locally) and Gemini or Anthropic for vision tasks (best quality). The pipeline config is saved to .clawd-config.json and loaded on startup.

For the hackathon submission, Gemini is the primary provider — @google/genai for the TypeScript desktop agent and google-genai for the Python browser agent. Vertex AI mode is supported for cloud deployments.

Challenges We Faced

The "blind clicking" problem. Early versions of the browser agent would sometimes click coordinates that looked right in the screenshot but were slightly off in the actual browser due to scaling and DPI differences. We fixed this by normalizing coordinates relative to the Playwright viewport size and adding a cursor indicator in the frontend so users can see exactly where the agent is clicking.

Context window management. Long browser sessions accumulate a lot of screenshots. Sending all of them to Gemini on every iteration would be prohibitively expensive and slow. The screenshot pruning strategy — keeping only the last 3 screenshots and summarizing older ones as text — was the right balance between context retention and cost.

Thread safety in the Python server. The WebSocket handler runs in an async event loop, but the Playwright browser and Gemini client calls are blocking. Getting the threading model right — async WebSocket handler, queue-based worker thread, run_coroutine_threadsafe for sending messages back — took several iterations to get stable.

The 5-layer pipeline ordering. Deciding which layer handles which task isn't always obvious. We went through many iterations of the routing logic before settling on the current approach: browser CDP first, then pattern matching, then accessibility tree, then vision. The key insight was that the accessibility tree is almost always faster and cheaper than a screenshot, and it's surprisingly capable for most UI tasks.

Cross-platform desktop control. Windows uses PowerShell for accessibility queries. macOS uses JXA (JavaScript for Automation) and System Events. Linux has neither. We ended up with platform-specific script directories (scripts/mac/) and a runtime check that routes to the right implementation. Linux falls back to browser-only mode.

What We Learned

Vision is the fallback, not the foundation. The instinct when building a visual agent is to route everything through the vision model. That's wrong. Vision is expensive and slow. Build the cheap layers first — pattern matching, accessibility trees, keyboard shortcuts — and use vision only when they fail. Your users will notice the difference.

The accessibility tree is underrated. Most developers don't think about the OS accessibility tree as an automation primitive. But it's a structured, real-time representation of every UI element on screen, with labels, roles, and positions. For a huge range of tasks, it's more reliable than a screenshot and orders of magnitude cheaper.

Streaming makes agents feel alive. Sending screenshots and reasoning updates to the frontend in real time — rather than waiting for the task to complete — fundamentally changes how the agent feels to use. Users can see the agent thinking. They can intervene if it's going wrong. It transforms a black box into a collaborator.

Safety gates build trust. The confirmation flow for risky actions isn't just a safety feature — it's a trust-building mechanism. When users see the agent pause and ask "I'm about to send this email, confirm?" they feel in control. That feeling of control is what makes people comfortable giving an AI agent access to their desktop.

What's Next

TaskPilot is one of three projects our team built for the Gemini Live Agent Challenge. We also built:

A real-time visual companion (Live Agents category) — an agent you can talk to naturally that sees through your camera, grounded in Cloud Vision, Document AI, and Natural Language API
A 3D interactive history explorer (Creative Storyteller category) — a globe you can explore, clicking locations to get rich interleaved historical narratives with Imagen-generated imagery and Cloud TTS narration

Each project targets a different category, but they all share the same philosophy: use the right tool for the right job, ground AI reasoning in structured data, and build experiences that feel genuinely useful rather than impressive demos.

Try It Yourself

GitHub: https://github.com/tarinagarwal/task-pilot
Demo Video: https://vimeo.com/1174159668?share=copy&fl=sv&fe=ci

Built with Gemini (@google/genai, google-genai), Playwright, Google Cloud Run, Artifact Registry, Cloud Speech-to-Text, Cloud Firestore, Terraform, Electron, TypeScript, Python, and Express.

Created for the Gemini Live Agent Challenge. #GeminiLiveAgentChallenge

DEV Community: Sarthak Rawat

Retrieval-Augmented Generation, Explained From First Principles

The problem RAG is trying to solve

RAG, fine-tuning, and long context solve different problems

The RAG pipeline

1. Indexing: prepare the knowledge

Load

Parse and clean

Chunking

Embeddings

Dense and sparse representations

What happens inside vector search?

HNSW: navigating a graph

IVF: search the relevant regions

Product quantization: make the vectors smaller

Why ANN recall becomes an engineering concern

Retrieval: why "just use vector search" is not enough

Vocabulary mismatch

Exact terms matter too

Query and document lengths differ

Redundancy

Hybrid retrieval: combine complementary searches

Query rewriting and decomposition

HyDE and multi-query retrieval

Reranking: retrieve broadly, then judge carefully

Diversity matters too

The generation stage: context is evidence, not decoration

More context is not automatically better

Citations

What happens when the answer isn't in the documents?

Conversational RAG: the query is not always the query

When RAG gives a bad answer: debug the pipeline, not just the model

How do you know whether a RAG system actually works?

The RAG triad

1. Context relevance

2. Faithfulness

3. Answer relevance

Retrieval metrics

RAGAS and LLM-as-judge

When basic RAG isn't enough

Self-RAG: should I retrieve, and did retrieval help?

Corrective RAG: don't trust the retriever blindly

GraphRAG: when relationships matter

Agentic RAG: what should the system do next?

Four different responses to four different problems

RAG in production is a different problem

Freshness

Latency

Cost

Security: the documents themselves can attack the system

Multimodal RAG

Observability: if you can't see the pipeline, you can't debug it

RAG is not always the right answer

Designing a RAG system from requirements

1. Understand the use case

2. Design ingestion

3. Design retrieval

4. Design generation

5. Design evaluation and operations

A practical example: designing RAG for a legal firm

What changes when the system grows?

The deeper lesson: RAG is a system of tradeoffs

RAG in one picture

Final takeaway

Multithreading in Java: a guide to actually understanding it

What multithreading actually is

Thread lifecycle, states, and the methods that matter

Race conditions: the counter that doesn't count

synchronized, monitors, and reentrancy

wait(), notify(), and coordinating threads

The Java Memory Model and volatile

Explicit locks: ReentrantLock, ReadWriteLock, StampedLock

Atomic variables and compare-and-swap

Deadlock, livelock, and starvation: a bank transfer gone wrong

The Executor framework and thread pools

Callable, Future, and CompletableFuture

Concurrent collections, and building a blocking queue by hand

Coordination utilities: CountDownLatch, CyclicBarrier, Semaphore

Virtual threads: what changed in Java 21

Producer-consumer, done properly

`synchronized`, monitors, and reentrancy

`wait()`, `notify()`, and coordinating threads

The Java Memory Model and `volatile`

Explicit locks: `ReentrantLock`, `ReadWriteLock`, `StampedLock`

`Callable`, `Future`, and `CompletableFuture`

Coordination utilities: `CountDownLatch`, `CyclicBarrier`, `Semaphore`