DEV Community: Damir Karimov

Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking

Damir Karimov — Wed, 12 Aug 2026 11:34:40 +0000

The first two parts of this series covered why production RAG systems fail and how the quality of the data foundation directly affects everything that comes after it. We looked at document ingestion, parsing, chunking, and metadata design—the layers responsible for turning raw information into something a retrieval system can actually work with.

But even perfectly processed documents are useless if retrieval cannot find the right information.

In this third part, we'll move into the retrieval layer itself. We'll look at why vector search alone is often insufficient, how semantic and lexical search complement each other, and how reranking can turn a large set of possible matches into a small set of highly relevant documents. We'll also cover query optimization, metadata filtering, and context compression—key techniques for building retrieval pipelines that perform reliably on real-world queries.

Production RAG Architecture Series

✅ Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search
✅ Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design
Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking (you are here)
Scaling RAG Systems: Production Architecture, Performance, and Cost Optimization
Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns

Chapter 7 — Embeddings

Embeddings are not magic.

They are coordinates.

That is the whole trick.

A piece of text goes in, a vector comes out, and now similar meanings sit close to each other in space.

If chunking decides what the system sees, embeddings decide how it remembers it.

That sounds abstract until you try to build retrieval on top of it. Then it becomes the center of the whole system.

What an embedding really is

Imagine a map.

On that map:

“dog” sits near “wolf”.
“invoice” sits near “payment”.
“upgrade” sits near “billing policy”.
“password reset” sits somewhere else.

The model is not understanding meaning the way a human does. It is learning a geometry where related things end up near each other. That geometry is what retrieval uses later.

And that is why embeddings matter so much.

If the geometry is good, retrieval feels smart.

If the geometry is bad, everything downstream starts guessing.

Why this is not enough

This is where people usually make the first mistake.

They think:

“If I use a good embedding model, retrieval will work.”

It won’t.

A good embedding model can only work with the text you give it. If the chunk is messy, too broad, too short, or stuffed with unrelated ideas, the vector will still be messy. Just in a more expensive way.

A bad chunk becomes a bad vector.

A bad vector becomes a bad candidate.

A bad candidate becomes a confident wrong answer.

A concrete example

Take these chunks:

1. Active invoices must be closed before upgrading.
2. Customers can upgrade from Professional to Enterprise.
3. How to reset your password.
4. Downgrading is allowed only if no active trials exist.

A decent embedding model should understand that 1 and 2 belong near upgrade-related questions, while 3 is clearly off in another part of the world.

That sounds obvious, but in real systems it gets messy fast.

Because now you have:

legal docs,
support docs,
product policies,
release notes,
tables,
code snippets,
and old versions of the same document all mixed together.

At that point embeddings are not a detail anymore.

They are the shape of the search space.

What makes a good embedding model

A good model for production should:

understand your language,
behave well on short queries,
not collapse technical terms into generic similarity,
and work on your actual domain, not just “general text.”

A model that is decent for blog posts may be weak for:

policy documents,
multilingual corpora,
technical manuals,
product docs with version numbers,
or support data full of exact identifiers.

So the real question is not “which embedding model is popular?”

The real question is “which model gives me the right geometry for my corpus?”

Example code

python

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

chunks = [
    "Active invoices must be closed before upgrading.",
    "Customers can upgrade from Professional to Enterprise.",
    "How to reset your password.",
    "Downgrading is allowed only if no active trials exist."
]

vectors = model.encode(chunks, normalize_embeddings=True)

def cosine(a, b):
    return float(np.dot(a, b))

query = "Can Enterprise customers upgrade directly from Professional while keeping active invoices?"
query_vector = model.encode([query], normalize_embeddings=True)[0]

ranked = []
for chunk, vector in zip(chunks, vectors):
    score = cosine(query_vector, vector)
    ranked.append((chunk, score))

ranked.sort(key=lambda x: x[1], reverse=True)

for chunk, score in ranked:
    print(f"{score:.4f} | {chunk}")

This is the smallest possible version of the idea.

Query becomes a vector.

Chunk becomes a vector.

Similarity becomes a number.

The number is not truth.

It is only a signal.

But in a good system, that signal is useful enough to move the right chunk to the top.

Why chunk shape changes embedding quality

A short query and a long chunk do not behave the same way.

A query like:

“Enterprise upgrade active invoices”

is compact and vague.

A chunk like:

“Customers can upgrade from Professional to Enterprise. Active invoices must be closed before upgrading. Contact billing if invoices remain open.”

contains multiple ideas.

The embedding becomes a compressed summary of all of that. If the chunk contains too many unrelated ideas, the vector turns into an average of everything, which is another way of saying it gets blurrier.

That is why embeddings and chunking are inseparable.

You cannot fix one without thinking about the other.

The domain problem

General embeddings are often good enough to impress in demos. Production is where they start revealing their limits.

A support system might need to understand:

plan names,
billing states,
status codes,
product tiers,
policy phrases,
internal jargon.

A generic model may know the words, but not the importance of those words in your system.

That is why evaluation on real queries matters.

Not benchmark queries.

Your queries.

The real lesson

Embeddings are not magic meaning detectors.

They are a way to build a space where retrieval can do its job.

If the space is designed well, the system can find the right things.

If the space is noisy, the retriever will still return something plausible, and plausible is often the most dangerous kind of wrong.

That is the entire game.

Chapter 8 — Hybrid Search

Vector search is good at meaning.

Keyword search is good at precision.

Production needs both.

That is the whole chapter.

If you only use embeddings, the system understands the idea of the query but can miss the exact phrase that actually matters. If you only use keywords, the system catches exact matches but misses the intent behind the question. Hybrid search exists because real users do both things at once.

Why vector search is not enough

Vector search is great when a person asks naturally.

“How do I upgrade my plan?”

That kind of question has room for interpretation. The model can infer the intent even if the wording is loose.

But then the user asks something like:

“Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?”

Now exact words matter.

Professional.
Enterprise.
active invoices.

A vector model may understand the general billing theme, but it can still miss the exact policy sentence because the answer depends on precise terms, not just conceptual similarity.

That is where pure semantic retrieval starts lying politely.

Why keyword search is not enough

Now flip the problem.

A keyword system is brilliant when the query contains exact tokens.

If the question includes:

product names,
version numbers,
error codes,
clause IDs,
policy names,
exact phrases,

then BM25 or another lexical retriever often finds the right passage instantly.

But if the user says:

“Can a customer move to the top tier if they still owe money?”

a pure keyword search may fail because the document says:

“Active invoices must be closed before upgrading.”

That is the same idea, but not the same wording.

So keyword search is precise, but not smart.

Vector search is smart, but not precise enough.

The answer is both

Hybrid search is not some fancy optimization.

It is the basic admission that no single retrieval signal is enough.

The flow usually looks like this:

Query
  ↓
Vector Search
  ↓
Keyword Search
  ↓
Fuse Results
  ↓
Rerank
  ↓
Send to LLM

The idea is simple:

semantic retrieval finds the concept,
keyword retrieval finds the exact phrase,
fusion combines the strengths,
reranking picks the best final candidates.

A real example

Take this query:

“Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?”

Vector search might return:

billing policy chunks,
plan upgrade chunks,
invoice-related chunks.

Keyword search might return:

exact mention of “Professional”,
exact mention of “Enterprise”,
exact mention of “active invoices”.

If you merge both lists, suddenly the system has a much better chance of building the full answer instead of just a vaguely related one.

That is the difference between “sounds right” and “is right.”

The fusion problem

The tricky part is that vector scores and BM25 scores do not live on the same scale.

You cannot just add them blindly and hope the universe respects your optimism.

That is why production systems use score fusion methods like:

weighted sum,
rank-based fusion,
Reciprocal Rank Fusion.

The exact method matters less than the principle:

do not force two different ranking systems to pretend they are the same thing.

RRF in plain English

Reciprocal Rank Fusion is popular because it rewards documents that rank well in both systems without caring too much about score scale.

A simple version looks like this:

def rrf_score(rank, k=60):
    return 1 / (k + rank)

def fuse_rrf(vector_ranked, bm25_ranked, k=60):
    scores = {}

    for rank, item in enumerate(vector_ranked, start=1):
        scores[item["id"]] = scores.get(item["id"], 0) + rrf_score(rank, k)

    for rank, item in enumerate(bm25_ranked, start=1):
        scores[item["id"]] = scores.get(item["id"], 0) + rrf_score(rank, k)

    ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
    return ranked

The point is not the exact formula.

The point is that the system stops trusting one retriever too much.

Metadata comes first

Hybrid search should not run across everything in the universe.

Before you search, you usually want to narrow the field:

language = en
document_type = policy
version = latest
department = billing

That way the retrievers are not wasting time on content that should never have been considered in the first place.

This matters because a good production system is not just about finding more.

It is about finding less, but better.

Example code

def hybrid_retrieve(query, vector_index, bm25_index, metadata_filters=None, top_k=10):
    vector_results = vector_index.search(query, top_k=50, filters=metadata_filters)
    bm25_results = bm25_index.search(query, top_k=50, filters=metadata_filters)

    fused = fuse_rrf(vector_results, bm25_results, k=60)

    top_candidate_ids = [item_id for item_id, _ in fused[:50]]
    return top_candidate_ids[:top_k]

That is the shape of the system:

search twice,
fuse,
narrow,
then rerank later.

Why hybrid search feels more natural

Hybrid search works because people do not ask questions in one pure mode.

Sometimes they say:

“upgrade plan”
“active invoices”
“GPT-4.1”
“ERR-5027”

Sometimes they say:

“what happens if I still owe money?”
“how do I move to a higher tier?”
“does this apply to old versions?”

They mix exact terms and fuzzy intent in the same sentence.

Hybrid search is basically the system saying:

“Fine. I’ll handle both.”

Chapter 9 — Reranking

Retrieval finds candidates.

Reranking chooses the one that actually deserves to survive.

That distinction sounds small until you build a real system and realize that the first-stage retriever is often good at finding the right neighborhood, but not good enough at choosing the right house. It gives you the right area. The reranker decides which door matters.

Why reranking exists

A query like this:

“Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?”

usually produces a decent shortlist from hybrid search. But shortlist is not answer quality. You might get:

one chunk about upgrading,
one chunk about invoices,
one chunk about billing support,
one chunk about a related policy,
and one chunk that is technically similar but not actually useful.

That is normal. Retrieval is designed to cast a wide net.

Reranking is what narrows that net into something the LLM can trust.

The core problem

Vector search and BM25 are fast.

Cross-encoder reranking is slower.

So production systems split the work:

Query
  ↓
Hybrid Search
  ↓
Top 20–50 candidates
  ↓
Cross-Encoder Reranker
  ↓
Top 3–5 chunks
  ↓
LLM

This is the standard retrieve-then-rerank shape because it balances speed and precision. The first stage optimizes recall. The second stage optimizes correctness.

Why the first ranking is not enough

The first retriever often returns something that is “close enough,” which is exactly the problem.

For example, imagine the shortlist contains:

1. "Customers can upgrade from Professional to Enterprise."
2. "Active invoices must be closed before upgrading."
3. "Billing support and payment history."
4. "How to reset your password."
5. "Downgrading is allowed only if no active trials exist."

A human can instantly see that 1 and 2 matter most. But the retriever only sees statistical similarity. It knows what is related, not what is most answer-bearing.

That is the gap reranking closes.

How a cross-encoder thinks

A cross-encoder takes the query and the candidate chunk together and scores the pair as one unit.

That is different from embeddings.

A bi-encoder says: “these two texts look similar in space.”
A cross-encoder says: “this chunk answers this query better than the other chunk.”

That extra interaction is expensive, but it is much more precise.

This is why rerankers are usually the cheapest way to improve answer quality once retrieval is already decent. They do not fix bad retrieval. They fix bad ordering.

A simple example

Suppose the query is:

“Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?”

And the retrieved chunks are:

“Customers can upgrade from Professional to Enterprise.”
“Active invoices must be closed before upgrading.”
“How to reset your password.”
“Downgrading is allowed only if no active trials exist.”

A reranker will likely push the first two to the top because they jointly answer the question. The third is irrelevant. The fourth is related but not the right policy branch.

That is the difference between:

finding similar text,
and finding the most useful text.

Example code

Here is the shape of a basic cross-encoder reranking step:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query, chunks, top_n=5):
    pairs = [[query, chunk["text"]] for chunk in chunks]
    scores = reranker.predict(pairs)

    ranked = sorted(zip(chunks, scores), key=lambda x: x[1], reverse=True)
    return [chunk for chunk, score in ranked[:top_n]]

That is the essential idea:

retrieve many,
score each query-chunk pair,
keep the best few.

Why reranking is so valuable

Reranking improves the part of the pipeline that users actually feel.

It helps when:

chunks are semantically close but not equally useful,
the corpus contains overlapping policies,
the query is specific,
exact answer selection matters,
the system has too much context noise.

In practice, reranking is one of the highest ROI improvements in production RAG because it upgrades quality without forcing you to rebuild everything else.

The hidden benefit

Reranking also makes debugging easier.

If retrieval looks good but the answer is still wrong, the problem may be:

bad reranking,
bad chunking,
or bad context construction.

If retrieval itself is weak, reranking cannot save it.

That is important. Reranking is not a miracle layer. It is a refinement layer.

Where reranking sits in the architecture

The right mental model is:

Retrieval finds enough candidates.
Reranking decides which candidates are worth using.
Generation turns those candidates into an answer.

If the first stage is the net, the reranker is the hand that chooses the fish you actually keep.

The Core Idea

Retrieval is not just about finding text that looks similar to a user's query. A production RAG system must determine which information is actually relevant, which sources should be trusted, and which results should be excluded before they ever reach the LLM.

Vector similarity provides semantic relevance, but it does not provide enough control on its own. Hybrid search, metadata filtering, and reranking work together to narrow a large candidate set into the small amount of context that the model actually needs.

That is the core idea behind production retrieval: don't just retrieve more information—retrieve the right information, in the right order, for the right query.

Continue the Series

This article focused on the retrieval layer of a production RAG system: embeddings, hybrid search, query optimization, reranking, and context compression. Together, these techniques help turn a large set of possible matches into a smaller, more relevant context for the LLM.

But a retrieval pipeline that works well on a small dataset can behave very differently when the system needs to handle millions of documents, concurrent users, strict latency requirements, and growing infrastructure costs.

In the next article, we'll move from retrieval quality to production scale. We'll explore how to design RAG architectures that remain fast, reliable, and cost-efficient as the amount of data and traffic grows.

Next up:

Part 4 — Scaling RAG Systems: Production Architecture and Performance Optimization

We'll cover:

Large-scale RAG architecture
Scaling ingestion and retrieval pipelines
Vector database performance and indexing
Caching and latency optimization
Async processing and background workers
Cost optimization
Designing RAG systems for millions of documents and concurrent users

By the end of this series, you'll have a complete engineering framework for designing, building, scaling, and evaluating production-grade Retrieval-Augmented Generation systems.

Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design

Damir Karimov — Wed, 05 Aug 2026 08:47:21 +0000

In the first article, we explored why many RAG systems fail in production and established a key principle: retrieval quality determines answer quality. We also introduced the architecture behind production-grade RAG systems and explained why a simple "embeddings + vector database + LLM" pipeline is rarely enough.

In this second part, we'll move one step earlier in the pipeline—to the moment a document first enters your system. We'll examine how documents should be ingested, cleaned, parsed, chunked, and enriched with metadata before a single embedding is generated. These decisions form the foundation of every production RAG system and often have a greater impact on retrieval quality than the choice of embedding model or LLM.

Production RAG Architecture Series

✅ Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search
Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design (you are here)
Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking
Scaling RAG Systems: Production Architecture and Performance Optimization
Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns

Chapter 4 — Ingestion: Where Retrieval Really Begins

In production, retrieval does not begin when the user asks a question. It begins the moment a document enters your system. If ingestion is weak, every later step inherits the damage, and no amount of clever prompting will fully recover it.

This is why ingestion is not a “PDF parsing task.” It is the process of turning messy human documents into structured, searchable, trustworthy knowledge.

Why ingestion matters

A document is rarely just plain text. A contract has sections, tables, headers, footers, page numbers, and sometimes scanned signatures. A handbook may contain headings, code blocks, bullet lists, and embedded screenshots. A policy document may look readable to a person but be almost useless to a machine if the structure is flattened too early.

If you simply extract all text and dump it into a chunker, you create noise. The model then sees fragments of meaning instead of complete ideas. That is how a document with a clear rule like “active invoices must be closed before upgrading” can turn into a retrieval result that feels relevant but answers the wrong thing.

What good ingestion does

A good ingestion pipeline does four things:

It preserves structure.
It removes noise.
It extracts metadata.
It prepares the document for retrieval.

That sounds obvious, but most weak systems do only one of these, and often badly. They extract text, maybe split it into chunks, and then assume the rest will work itself out. In practice, ingestion is where the system either becomes searchable knowledge or remains a pile of text.

A simple mental model

Think of ingestion like preparing ingredients before cooking.

You do not throw whole vegetables, packaging, and labels into the pan and hope the meal will sort itself out. You wash, peel, cut, sort, and label things first. In the same way, a document should be cleaned, structured, and annotated before it is indexed.

Here is the basic shape of the pipeline:

Upload 
    ↓     
Object Storage 
    ↓ 
Parser 
    ↓ 
Cleaner 
    ↓ 
Structure-aware Chunker 
    ↓ 
Metadata Extractor 
    ↓ 
Embeddings 
    ↓ 
Indexes

Each step exists because raw documents are messy and retrieval systems are picky.

Parsing is not extraction

Parsing is about understanding format, not just getting text out. A PDF parser should know whether the document is scanned or digital, whether the content is laid out in columns, whether tables are visually important, and whether the file contains repeated headers or footers. An HTML parser should preserve semantic tags. A Markdown parser should keep headings, lists, and code blocks intact.

If the parser is dumb, everything after it gets worse.

Cleaning is not optional

After parsing, the document usually still contains junk. You may see:

page numbers,
repeated company names,
“confidential” banners,
broken line wraps,
duplicated paragraphs,
OCR errors.

If you do not remove these, they will be embedded, stored, and later retrieved as if they mattered. That is one of the quietest ways to ruin search quality.

A clean ingestion pipeline removes noise without deleting meaning. That balance matters. Over-cleaning can damage content, while under-cleaning pollutes retrieval.

Structure preservation

Structure is what makes a document usable. Headings tell you what a section is about. Tables encode relationships. Lists show hierarchy. Code blocks preserve exact syntax. Losing that structure is often more harmful than losing a few words.

For example, these two representations are not equivalent:

Payment 
Terms Invoices must be paid within 30 days. 
Failure to pay may result in service suspension.

and:

Invoices must be paid within 30 days. 
Failure to pay may result in service suspension.

The second one is technically readable, but the first one is much more useful because the heading gives the chunk a clear semantic frame. That frame helps retrieval and helps the LLM explain the answer later.

Metadata begins here

Metadata is not something you attach at the end. It should be born during ingestion.

Useful metadata often includes:

document type,
title,
section,
page number,
language,
version,
source system,
department,
creation time,
last updated time.

This metadata is what lets the system answer questions like:

“Show me the latest policy.”
“Search only English documents.”
“Find finance documents.”
“Ignore deprecated versions.”
“Return the clause from page 12.”

Without metadata, everything becomes a vague semantic soup.

A practical example

Suppose you ingest this text:

Title: Pricing Policy

Section: Upgrading Plans
Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.

Section: Downgrading Plans
Downgrading is allowed only if no active trials exist.

A weak pipeline might store it as one long blob. A better pipeline creates structured records like this:

{
  "text": "Customers can upgrade from Professional to Enterprise. Important: Active invoices must be closed before upgrading.",
  "metadata": {
    "title": "Pricing Policy",
    "section": "Upgrading Plans",
    "document_type": "policy",
    "language": "en",
    "version": "2.1"
  }
}

Now the retriever can filter, rank, and cite this chunk much more intelligently.

OCR and scanned documents

Scanned documents are where ingestion often breaks in production. If text exists only as pixels, you need OCR before anything else can happen. And OCR is not just a fallback feature for edge cases. In many real organizations, a large share of useful knowledge lives in scanned forms, signed contracts, invoices, and old PDFs.

If OCR is poor, the system may miss critical words or mix table cells together. If layout is ignored, the model may see nonsense like one long broken sentence or a table flattened into unreadable text. That is why good ingestion treats OCR and layout extraction as first-class concerns.

The main failure pattern

The most common failure pattern is this:

Document enters the system.
Text is extracted too early.
Structure is lost.
Chunking splits the wrong boundaries.
Metadata is missing or shallow.
Retrieval returns “relevant” but incomplete context.
The LLM answers confidently with the wrong interpretation.

That failure often looks like an AI problem, but it is really an ingestion problem.

What to aim for

Your ingestion pipeline should produce chunks that are:

readable on their own,
tied to a known section or source,
enriched with metadata,
clean enough to embed,
and traceable back to the original document.

If a chunk cannot tell you where it came from and why it exists, it is probably not ready for production.

The core idea

Ingestion is not a support task for retrieval. It is the foundation of retrieval.

That is why production RAG systems spend so much effort on parsing, cleaning, structure, and metadata before they ever touch embeddings. Once that foundation is solid, everything downstream becomes easier, cheaper, and more trustworthy.

Chapter 5 — Chunking: The Heart of Retrieval Quality

Chunking is where most RAG systems quietly fall apart. You can have a solid model, a fast vector database, and a polished prompt, but if the chunks are badly shaped, retrieval will still miss the point.

The reason is simple: retrieval does not search for “truth.” It searches for pieces of text that look useful. If those pieces are incomplete, noisy, or cut in the wrong place, the whole system starts answering with confidence and precision in all the wrong directions.

Why chunking matters

Think about reading a manual where every page was cut into random strips and shuffled. You could still recognize some words, but you would not trust the result. That is exactly what happens when documents are split by token count only.

A good chunk is not just “small enough.” It should be:

semantically coherent,
self-contained enough to understand,
aligned with document structure,
and useful on its own when retrieved.

If a chunk needs three other chunks just to make sense, retrieval quality drops fast.

The wrong way to chunk

The most common beginner approach is fixed-size splitting.

1000 tokens
↓
cut anywhere
↓
chunk

It looks clean in code and terrible in practice. A sentence may get broken in half. A table row may be separated from its header. A section title may end up in one chunk while the content appears in another. Then the retriever sees fragments instead of ideas.

For example:

Original:
"Active invoices must be closed before upgrading from Professional to Enterprise."

Bad chunk 1:
"Active invoices must be closed before upgrading from Professional"

Bad chunk 2:
"to Enterprise."

Neither chunk is ideal. One is incomplete, the other is meaningless. The model may still answer, but it will be guessing.

Fixed-size chunking

Fixed-size chunking is easy to implement and easy to explain, which is why it appears in so many tutorials. It is fine for quick experiments, but it should rarely be your final choice.

Good for:

raw plain text,
prototypes,
quick baselines.

Bad for:

contracts,
docs with headings,
tables,
code-heavy documents,
anything where structure matters.

The problem is not just correctness. Fixed-size splitting also creates retrieval noise. Because chunks are arbitrarily cut, the vector representation becomes less focused. That means the search engine has to work harder just to recover meaning that was destroyed during preprocessing.

Sliding window chunking

Sliding window chunking is the first small improvement. Instead of cutting once and moving on, you keep overlap between adjacent chunks.

Chunk 1: [0..500]
Chunk 2: [400..900]
Chunk 3: [800..1300]

This helps preserve context at the boundaries. If an important sentence gets cut in one chunk, it may still appear in the next one. That alone can significantly improve retrieval.

But overlap is not magic. If the source structure is bad, sliding windows just give you two slightly redundant bad chunks instead of one. It reduces damage; it does not solve the design problem.

Structure-aware chunking

A better approach is to chunk according to the document itself.

If the source is Markdown, use headings. If it is HTML, use semantic tags. If it is a contract, use clauses and sections. If it is a handbook, use chapters and paragraphs.

That gives you chunks like this:

Section: Upgrading Plans

Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.

This is much stronger than a blind token slice because the chunk carries its own context. When the retriever finds it, the LLM sees something that already resembles an answer-ready unit.

Semantic chunking

Semantic chunking goes one step further. Instead of splitting based only on formatting, it tries to split by meaning. The system looks for topic shifts, idea boundaries, or sentence groups that belong together.

This is especially useful when documents are less formal:

product documentation,
internal notes,
policy text,
long articles,
mixed-format knowledge bases.

A semantic chunk should feel like a paragraph a human would naturally quote. If you would not send it to a colleague as a useful excerpt, it is probably not a good chunk.

Parent-child chunking

One of the most effective production patterns is parent-child chunking.

The idea is simple:

search with small child chunks,
answer with larger parent chunks.

Why this works:

small chunks are easier to match precisely,
large chunks provide enough surrounding context for generation.

Example:

Parent chunk:
Section: Upgrading Plans
Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.
If invoices are open, contact billing.

Child chunks:
1. Customers can upgrade from Professional to Enterprise.
2. Important: Active invoices must be closed before upgrading.
3. If invoices are open, contact billing.

The child chunks help retrieval find the relevant idea. The parent chunk gives the model the full answer. This reduces both hallucination and context loss.

Hierarchical chunking

Hierarchical chunking is what happens when you stop thinking in one layer.

Instead of one flat list of chunks, you keep multiple levels:

document,
section,
paragraph,
sentence.

That gives you more control. A broad query can retrieve a section. A precise query can retrieve a paragraph. A citation can point to the exact sentence.

This is especially useful in large corpora where one document may contain many unrelated topics. If you only use flat chunks, the system may know the right document but still miss the right passage.

Recursive chunking

Recursive chunking is a practical compromise. You split by large boundaries first, then smaller ones if needed.

For example:

split by headings,
then paragraphs,
then sentences,
then tokens if absolutely necessary.

This keeps structure intact whenever possible and only falls back to smaller splits when the document is too large. It is a solid default for many production systems because it behaves more like a human editor than a token cutter.

Contextual chunking

Sometimes a chunk is too short to stand alone, even if it is semantically clean. In that case, you add context around it.

Example:

Document title: Pricing Policy
Section: Upgrading Plans

Chunk text:
Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.

This extra context makes the chunk much easier to understand at retrieval time. It also helps the LLM produce better grounded answers because the chunk now carries its own label.

When chunking goes wrong

Chunking failures usually show up in very human ways:

the answer is almost right but misses one rule,
the LLM cites the wrong section,
two different policies get blended into one answer,
a table becomes unreadable,
a “yes” answer appears when the real answer is “yes, but only if...”.

That is why chunking is not a preprocessing detail. It is an information design problem.

Practical guidance

If you are building a real production system, start like this:

preserve structure before splitting,
keep headings with their content,
avoid arbitrary cuts inside tables or code blocks,
use overlap only when needed,
test chunks with real queries, not just with sample text.

A good chunk should answer one natural question or support one natural fact. If it feels like a random slice of text, it probably is.

The main idea

Chunking is not about making text smaller. It is about making knowledge retrievable.

That sounds simple, but it is the difference between a chatbot that sounds smart and a system people actually trust.

Chapter 6 — Metadata: The Difference Between “Similar” and “Correct”

Metadata is where production RAG stops being a clever text search system and starts becoming a trustworthy product. Without metadata, the system can only say, “this looks related”; with metadata, it can say, “this is the right document, from the right version, for the right user, in the right context.”

Why metadata matters

Imagine a supermarket search.

A customer asks for “red dry wine under 20 euros.” If you only use semantic similarity, you may get wines that are red, sweet, expensive, or merely described in a similar style. If you also apply structured filters, you immediately rule out the wrong candidates before retrieval gets expensive or confused. Metadata turns vague similarity into precise constraint handling.

RAG behaves the same way. A user often does not want any semantically related chunk. They want the latest policy, the right language, the correct region, the right team, or the current version of a document. Metadata lets the system narrow the world before it starts ranking chunks.

What metadata actually is

Metadata is not extra decoration. It is structured information attached to documents and chunks so the system can reason about them later.

Useful metadata usually includes:

document title,
document type,
section or heading,
page number,
language,
version,
source system,
department,
created and updated timestamps,
access level,
tags or topic labels.

At the document level, metadata helps you know what something is. At the chunk level, it helps you know where a passage came from and whether it should be considered for a particular query.

Why chunks need metadata too

A chunk without metadata is just floating text. It may be semantically meaningful, but it loses the context that makes retrieval reliable.

For example, this chunk:

Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.

is much more useful if it also carries:

{
  "title": "Pricing Policy",
  "section": "Upgrading Plans",
  "document_type": "policy",
  "language": "en",
  "version": "2.1"
}

Now the retriever can do more than similarity search. It can filter out old versions, ignore unrelated departments, and keep answers tied to the source section that actually matters.

Metadata is a retrieval control system

Think of retrieval as two decisions:

Which documents are even eligible?
Among those, which chunks are best?

Metadata answers the first question.

A user asking about “the latest finance policy for EMEA” does not need every chunk in your system. They need:

finance documents,
current version,
relevant region,
probably English,
probably policy, not support chat or engineering notes.

If you do this with pure vector similarity, the system has to discover all of that indirectly. That is slow, noisy, and unreliable.

Hard filters and soft filters

Not all metadata is equal.

Some metadata is hard:

tenant ID,
permissions,
access level,
document status,
version validity.

These are non-negotiable. If a document is superseded, deleted, or private, it should not be considered at all.

Other metadata is soft:

topic tags,
department,
region,
audience level,
freshness preference.

These improve relevance, but they should not cause the system to return nothing if they are slightly off.

A good production system treats hard filters as gates and soft filters as ranking hints.

Example: same question, different metadata

Suppose a user asks:

Can a customer upgrade from Professional to Enterprise with active invoices?

The system should not just search for similar wording. It should also know whether the answer belongs to:

a current pricing policy,
a deprecated policy,
a support article,
a legal document,
or an internal note.

Without metadata, all of these may compete in the ranking pool. With metadata, the search space becomes much smaller and much safer.

Query-time filtering

Metadata is most powerful when it is used at query time.

A good flow looks like this:

User Query
  ↓
Extract likely filters
  ↓
Apply hard constraints
  ↓
Search within allowed subset
  ↓
Rerank results
  ↓
Send best chunks to LLM

For example, if the query implies:

language = en
document_type = policy
version = latest

then those filters should shape the retrieval before the model sees anything.

That is one of the biggest reasons production RAG feels much better than tutorial RAG: it is not searching the whole world anymore. It is searching the right slice of the world.

Metadata-aware chunking

Metadata is also a chunking problem.

If a paragraph belongs to a section, the chunk should know that. If a table belongs to a page, the chunk should know that too. If a chunk comes from a source that is deprecated, it should carry that status.

Good chunk metadata often includes:

parent document ID,
section name,
page number,
hierarchy path,
extraction method,
language,
freshness state.

This helps in three ways:

filtering,
ranking,
citations.

Without this, the system may retrieve the right words but fail to explain where they came from.

A simple example schema

Here is a practical chunk representation:

{
  "id": "pricing_policy_p12_s3_c2",
  "text": "Active invoices must be closed before upgrading from Professional to Enterprise.",
  "metadata": {
    "title": "Pricing Policy",
    "section": "Upgrading Plans",
    "page": 12,
    "document_type": "policy",
    "language": "en",
    "version": "2.1",
    "status": "current",
    "department": "billing"
  }
}

This chunk is now useful for:

semantic search,
exact filtering,
citations,
debugging,
auditing.

What happens without metadata

Without metadata, retrieval tends to become “close enough” search. That sounds fine until the stakes rise.

Examples of failure:

a user gets an answer from an outdated policy,
the system returns the wrong region,
support sees a marketing document instead of a billing rule,
a legal question is answered from an internal note,
a multilingual corpus returns the wrong language.

The model may still generate a fluent answer, but fluency is not correctness.

The core idea

Metadata is not a nice-to-have. It is the thing that turns retrieval from approximate text similarity into controlled information access.

If chunking defines what a unit of knowledge looks like, metadata defines when and why that unit should be used.

That is why serious RAG systems treat metadata design as a first-class architecture decision, not a last-minute indexing detail.

Continue the Series

This article covered the foundation of every production RAG system: document ingestion, parsing, chunking, and metadata design. These preprocessing decisions determine how effectively your retrieval pipeline can find relevant information long before an LLM generates an answer.

In the next article, we'll move from document preparation to retrieval itself. We'll explore why vector search alone is not enough and how modern production systems combine multiple retrieval techniques to achieve significantly higher accuracy.

Next up:

Part 3 — Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking

We'll cover:

Embeddings and semantic representation
Hybrid search (vector search + BM25)
Query rewriting and optimization
Reranking with cross-encoders
Context compression strategies
Building a production retrieval pipeline

By the end of this series, you'll have a complete engineering framework for designing, building, scaling, and evaluating production-grade Retrieval-Augmented Generation systems.

Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search

Damir Karimov — Thu, 23 Jul 2026 12:12:04 +0000

Building a reliable RAG system is not about connecting an LLM to a vector database. It requires a complete architecture covering data ingestion, retrieval, ranking, evaluation, and production operations.

This article is the first part of a five-part series where we will explore how modern RAG systems are designed, why naive implementations fail, and what it takes to build AI search systems that work reliably in production.

Production RAG Architecture Series

Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search (you are here)
Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design
Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking
Scaling RAG Systems: Production Architecture and Performance Optimization
Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns

Chapter 1 — Why Most RAG Systems Fail in Production

Demo works. Production fails.

Building a Retrieval-Augmented Generation system isn't hard.

Building one that people can trust is.

You've probably seen hundreds of tutorials:

Documents → Split into Chunks → Generate Embeddings → Store in Vector DB → Similarity Search → Send Context to LLM → Answer

It looks beautifully simple.

You can build a working chatbot in 30 minutes with LangChain, LlamaIndex, or Haystack.

Upload a PDF.

Generate embeddings.

Store them.

Ask questions.

Done.

For a demo, that's enough.

For production, that's the beginning of your problems.

Imagine this.

You built an internal AI assistant for your company's documentation.

Demo:

User: "How do I reset my password?"

Assistant: "Go to Settings → Security → Reset Password. You'll receive an email with a link."

Perfect. Everyone applauds. Project approved.

Two weeks later, support starts using it.

User: "Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"

Assistant: "Yes."

Actual answer from the policy document:

"Active invoices must be closed before upgrading from Professional to Enterprise."

Nobody notices immediately.

Support gives wrong advice.

Customer tries to upgrade.

Billing rejects.

Support spends hours resolving.

Management asks:

"I thought AI was supposed to reduce support workload."

Nothing crashed.

Embeddings were correct.

Vector DB was running.

Similarity search returned relevant chunks.

LLM generated fluent text.

Every component worked.

Yet the system failed.

That is what makes production RAG difficult.

Failures rarely come from broken software.

They come from broken architecture.

The tutorial pipeline vs the real pipeline

Most tutorials focus on this:

Query
  ↓
Vector Search
  ↓
LLM
  ↓
Answer

Real production systems look more like this:

Ingestion (Offline)

  Upload
    ↓
  Object Storage (S3 / GCS)
    ↓
  OCR / Parser (PDF, HTML, Markdown)
    ↓
  Content Cleaning (headers, footers, page numbers)
    ↓
  Chunking (semantic + structure-aware)
    ↓
  Metadata Extraction (doc type, language, version, section)
    ↓
  Embedding Workers
    ↓
    ├─→ Vector DB
    └─→ BM25 / Keyword Index

----------------------------

Retrieval (Online)

  User Query
    ↓
  Query Rewriting / Expansion
    ↓
  Metadata Filters (language, department, version)
    ↓
  Hybrid Search (Vector + BM25)
    ↓
  Reranking (Cross-Encoder)
    ↓
  Context Compression
    ↓
  Prompt Builder
    ↓
  LLM
    ↓
  Answer + Citations

The language model is often the smallest part of the architecture.

But it receives almost all attention.

Retrieval is not solved

The biggest misconception:

"Retrieval is a solved problem. Just use a vector DB."

It isn't.

Think about the Library of Congress.

Naive approach:

Throw all books into a giant pile.
Ask someone to find the right one by reading random pages.
Eventually, they might find it.

This is basically what many beginner RAG systems do:

Split everything into fixed-size chunks.
Embed.
Retrieve top-k by cosine similarity.
Feed to LLM.

Professional librarian:

Before searching, they narrow it down:

Language?
Publication year?
Author?
Category?
Edition?
Print or digital?
Academic or popular?

Only after reducing millions of possibilities to a few hundred, they start semantic search.

Professional retrieval works the same way.

Goal is not to search better.

Goal is to search less.

Vector DB is an index, not a database

Another common mistake:

"Vector DB replaces traditional databases."

No.

A vector DB is not your source of truth.
Think about Google:
- Google doesn't own the internet.
- It indexes the internet.
- If Google servers disappear, websites still exist.

Same with RAG:

Your documents should live somewhere else:
- PostgreSQL
- S3 / GCS / Azure Blob
- SharePoint / Confluence / Notion
Vector DB only stores an efficient representation for fast retrieval.
Delete your vectors → you can rebuild them.
Delete your original documents → you lost everything.

Many production systems fail because developers treat Vector DB as permanent storage.

Retrieval starts at ingestion

Another mistake:

"Retrieval begins when the user submits a question."

No.

Retrieval begins the moment a document enters your system.

Imagine two companies.

Company A:

Employee uploads PDF.
System extracts plain text.
Splits every 500 tokens.
Generates embeddings.
Stores them.
Done.

Company B:

Same PDF arrives.

Before embeddings, system asks:

Is this real text or scanned images? (OCR needed?)
Does it contain tables?
Headers / footers?
Multiple languages?
Version numbers?
Duplicate pages?
References to other documents?
Sensitive information?
Structured sections (chapters, clauses)?

Only after understanding the document, indexing begins.

Which system produces better answers 6 months later?

Not the one with the better LLM.

The one with better ingestion.

RAG is an information retrieval problem

If there's one idea to remember:

RAG is not an AI problem.

RAG is an information retrieval problem that happens to use AI.

That changes everything.

Instead of asking:

"Which LLM should I use?"

You start asking:

How should documents be represented?
How should information be organized?
What makes one chunk better than another?
Which metadata should be extracted?
Which ranking strategy should be used?
How do we measure retrieval quality?
How do we know retrieval failed?

Those matter more than choosing GPT-5, Claude, or Gemini.

A brilliant LLM cannot generate knowledge it never received.

Garbage retrieval → garbage answers. Every time.

Chapter 2 — The Toy RAG (with real code and examples)

Now let's see what a toy RAG really looks like, and why it fails.

We'll use Python + sentence-transformers + chromadb (local vector DB).

No LangChain magic, just raw code, so you see what's happening.

We'll build:

A naive RAG (fixed-size chunks).
Show where it breaks.
Compare with a slightly better version (structure-aware chunks + metadata).

Scenario: Company Knowledge Base

Imagine you have a simple knowledge base:

pricing_policy.md
onboarding_guide.md
support_rules.md

Example content from pricing_policy.md:

# Pricing Policy

## Upgrading Plans

Customers can upgrade from Professional to Enterprise.

**Important:** Active invoices must be closed before upgrading.

If you have any open invoices, contact billing@example.com.

## Downgrading Plans

Downgrading is allowed only if:
- No active trials
- No pending invoices

And a user query:

"Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"

Step 1: Naive RAG with fixed-size chunks

from pathlib import Path
from typing import List
import chromadb
from chromadb.config import Settings
from sentence_transformers import SentenceTransformer

# 1. Load documents
documents_dir = Path("docs")
texts = []
for md_file in documents_dir.glob("*.md"):
    texts.append(md_file.read_text(encoding="utf-8"))

# 2. Naive chunking: fixed size
def naive_chunk(text: str, chunk_size: int = 500, overlap: int = 50) -> List[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

all_chunks = []
for text in texts:
    all_chunks.extend(naive_chunk(text))

# 3. Embeddings
embedder = SentenceTransformer("all-MiniLM-L6-v2")
chunk_embeddings = embedder.encode(all_chunks, convert_numpy=True)

# 4. Store in Chroma
chroma_client = chromadb.Client(Settings(
    chroma_db_impl="duckdb+parquet",
    persist_directory="./chroma_db"
))
collection = chroma_client.create_collection("naive_rag")

for i, chunk in enumerate(all_chunks):
    collection.add(
        ids=[f"chunk_{i}"],
        embeddings=[chunk_embeddings[i].tolist()],
        documents=[chunk]
    )

# 5. Query
def query_naive_rag(collection, query: str, top_k: int = 5) -> List[str]:
    query_emb = embedder.encode([query], convert_numpy=True)[0].tolist()
    result = collection.query(
        query_embeddings=[query_emb],
        n_results=top_k
    )
    return result["documents"][0]

query = "Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"
retrieved = query_naive_rag(collection, query, top_k=5)

for i, chunk in enumerate(retrieved, 1):
    print(f"Chunk {i}:\n{chunk}\n")

What you'll likely see:

Some chunks might contain:

"Customers can upgrade from Professional to Enterprise."
"Downgrading is allowed only if..."
Random fragments with "invoices" but not the right rule.

But the crucial sentence:

"Active invoices must be closed before upgrading from Professional to Enterprise."

might be:

Split across two chunks.
Buried in a chunk with unrelated text.
Not in top-k at all.

Then LLM will generate:

"Yes, they can upgrade."

Because the retrieved context doesn't say "No".

This is exactly how naive RAG fails.

Not because embedding is wrong.

Because chunking + retrieval are not designed for real questions.

Step 2: Slightly better RAG with structure-aware chunks + metadata

Now let's do it more thoughtfully.

We'll:

Split by Markdown headings.
Keep section title + content together.
Add metadata: doc_name, section, language.

import re
from typing import List, Dict

def structure_aware_chunk(text: str, doc_name: str) -> List[Dict]:
    chunks = []

    # Split by headings: # Section, ## Subsection
    pattern = r"(^#{1,2}\s+.+$)"
    parts = re.split(pattern, text, flags=re.MULTILINE)

    # parts: [before_first_heading, heading1, content1, heading2, content2, ...]

    current_heading = None
    current_content = []

    for i, part in enumerate(parts):
        if i == 0 and part.strip() == "":
            continue
        if re.match(pattern, part):
            # New heading
            current_heading = part.strip()
            current_content = []
        else:
            # Content
            if current_heading:
                current_content.append(part.strip())

            if current_heading and (i + 1 >= len(parts) or re.match(pattern, parts[i + 1])):
                # End of section
                content_text = "\n\n".join(current_content).strip()
                if content_text:
                    chunks.append({
                        "text": f"{current_heading}\n\n{content_text}",
                        "metadata": {
                            "doc_name": doc_name,
                            "section": current_heading,
                            "language": "en",
                        }
                    })
                    current_heading = None
                    current_content = []

    return chunks

all_chunks = []
for md_file in documents_dir.glob("*.md"):
    doc_name = md_file.name
    text = md_file.read_text(encoding="utf-8")
    all_chunks.extend(structure_aware_chunk(text, doc_name))

# Embeddings + store (similar to before, but with metadata)
chunk_embeddings = embedder.encode([c["text"] for c in all_chunks], convert_numpy=True)

collection2 = chroma_client.create_collection("better_rag")

for i, chunk in enumerate(all_chunks):
    collection2.add(
        ids=[f"chunk_{i}"],
        embeddings=[chunk_embeddings[i].tolist()],
        documents=[chunk["text"]],
        metadatas=[chunk["metadata"]]
    )

def query_better_rag(collection, query: str, top_k: int = 5) -> List[Dict]:
    query_emb = embedder.encode([query], convert_numpy=True)[0].tolist()
    result = collection.query(
        query_embeddings=[query_emb],
        n_results=top_k
    )
    # Combine text + metadata
    return [
        {"text": doc, "metadata": meta}
        for doc, meta in zip(result["documents"][0], result["metadatas"][0])
    ]

retrieved2 = query_better_rag(collection2, query, top_k=5)

for i, item in enumerate(retrieved2, 1):
    print(f"Chunk {i} (from {item['metadata']['doc_name']}, section: {item['metadata']['section']}):")
    print(item["text"])
    print()

What improves:

Chunks now preserve sections:
- "## Upgrading Plans" + full content.
Metadata tells you:
- Which document.
- Which section.
Retrieval more likely to find the exact section about upgrading + invoices.

Now context to LLM might look like:

## Upgrading Plans

Customers can upgrade from Professional to Enterprise.

Important: Active invoices must be closed before upgrading.

If you have any open invoices, contact billing@example.com.

Then LLM can answer:

"No. Active invoices must be closed before upgrading from Professional to Enterprise."

Same embedding model.

Same LLM.

But better chunking + metadata → better retrieval → better answer.

What this shows

Even in this tiny example:

Fixed-size chunks → fragmentation → wrong answer.
Structure-aware chunks + metadata → better context → correct answer.

In production, differences are even bigger:

Millions of documents.
Many languages.
Many versions.
Complex queries.

If you don't design ingestion, chunking, and metadata properly, no LLM will save you.

Chapter 3 — Every Production RAG Has Two Pipelines

If you've only followed tutorials, you probably think RAG is one pipeline:

Query
  ↓
Retrieve
  ↓
Generate
  ↓
Answer

That's enough for a demo.

It's catastrophic for production.

In any real system, you actually have two separate pipelines:

Offline (ingestion) pipeline – processes documents.
Online (query) pipeline – handles user requests.

They have completely different goals, constraints, and failure modes.

Designing them as one monolithic flow is one of the first mistakes that leads to production failures.

The Offline Pipeline: Where Documents Live Their "Slow Life"

The offline pipeline is what happens to documents before anyone ever asks a question.

Offline Pipeline (ingestion)

  Documents (PDF, HTML, Markdown, JSON)
    ↓
  Parser (structure-aware)
    ↓
  Cleaner (headers, footers, page numbers)
    ↓
  Chunker (semantic / structure-aware)
    ↓
  Metadata Extractor
    ↓
  Embedder
    ↓
  Indexer
    ├─→ Vector DB
    └─→ BM25 / Keyword Index

----------------------------

Online Pipeline (query)

  User Query
    ↓
  Query Processor (rewrite, expand)
    ↓
  Metadata Filters
    ↓
  Hybrid Search (Vector + BM25)
    ↓
  Reranker (Cross-Encoder)
    ↓
  Context Compression
    ↓
  Prompt Builder
    ↓
  LLM
    ↓
  Answer + Citations

What this pipeline does

Parser

*   Reads raw documents.

*   Extracts structure: headings, tables, lists, code blocks, paragraphs.

*   For PDFs: uses OCR if needed, understands layout.

*   For Markdown / HTML: preserves semantic tags.

Cleaner

*   Removes noise:

    *   Headers / footers ("Company", "Confidential").

    *   Page numbers.

    *   Watermarks.

    *   Repeated boilerplate text.

*   Normalizes whitespace, encoding, line breaks.

*   Fixes broken sentences or artifacts.

Chunker

*   Splits documents into meaningful chunks.

*   Ensures:

    *   Headings are not cut in the middle.

    *   Tables stay intact.

    *   Paragraph boundaries are respected.

*   Uses strategies like:

    *   Structure-aware chunking

    *   Semantic chunking

    *   Parent-child chunking

    *   (We'll cover these in detail in Chapter 5.)

Metadata Extractor

*   Extracts document-level metadata:

    *   `document_type`

    *   `language`

    *   `version`

    *   `department`

    *   `created_at`, `updated_at`

*   Extracts chunk-level metadata:

    *   `section` / `heading`

    *   `page`

    *   `chapter`

    *   `parent_id`

    *   `hierarchy_path`

Embedder

*   Computes embeddings for each chunk.

*   Optionally computes document-level embeddings too.

*   Uses models appropriate for:

    *   Domain (technical, legal, general).

    *   Language (English, Russian, multilingual).

Indexer

*   Inserts chunks into:

    *   Vector DB (dense embeddings).

    *   BM25 / keyword index (sparse).

    *   (Optionally) graph index (for knowledge graphs).

Key properties of the offline pipeline

It doesn't need to be fast.

Processing a document can take seconds or minutes.

Documents are not requested by users in real time.
It must be correct and robust.

If ingestion is wrong, retrieval will never be right.

Bad chunks, missing metadata, wrong parsing → garbage retrieval.
It can be batched or incremental.
- Batch: nightly jobs, reprocess entire corpus.
- Incremental: process only new/updated documents.
- Event-driven: triggered by upload events via queues.

This pipeline is where you decide:

How documents are represented.
How information is organized.
What metadata is available for filtering.
What kind of chunks the retriever will see.

If you ignore this pipeline and only focus on "query → retrieve → generate", you're building a demo, not a production system.

The Online Pipeline: Where Requests Live Their "Fast Life"

The online pipeline is what happens when a user asks a question.

Online Pipeline (query)

  User Query
    ↓
  Query Processor (rewrite, expand, normalize)
    ↓
  Metadata Filters (language, department, version, etc.)
    ↓
  Hybrid Search (Vector + BM25)
    ↓
  Reranker (Cross-Encoder)
    ↓
  Context Compression
    ↓
  Prompt Builder
    ↓
  LLM
    ↓
  Answer + Citations

What this pipeline does

Query Processor

*   Rewrites or expands the query:

    *   "upgrade plan" → "upgrade from Professional to Enterprise plan".

    *   Adds implicit context (user role, department).

*   Normalizes:

    *   Lowercases, removes noise.

    *   Handles typos, slang, abbreviations.

Metadata Filters

*   Applies filters based on:

    *   User's language.

    *   User's department / team.

    *   Document version (latest, stable, deprecated).

    *   Document type (policy, doc, article).

*   Reduces the candidate set before search.

Hybrid Search

*   Runs:

    *   Vector search (semantic similarity).

    *   BM25 / keyword search (exact term matching).

*   Combines scores:

    *   `final_score = α * vector_score + β * bm25_score`.

Reranker

*   Takes top-K candidates (e.g., 50–100).

*   Uses a cross-encoder to compute a more precise relevance score for each.

*   Sorts and picks the best N (e.g., 5–10).

Context Compression

*   Limits the number of chunks sent to the LLM.

*   Drops low-relevance chunks.

*   Optionally summarizes or extracts key sentences.

*   Ensures the prompt stays within token limits.

Prompt Builder

*   Constructs the final prompt:

    *   System instructions.

    *   Retrieved context.

    *   User query.

    *   Output format instructions (citations, confidence, etc.).

LLM

*   Generates the answer based on the prompt.

*   Ideally grounded in the retrieved context.

Answer + Citations

*   Returns:

    *   The final answer.

    *   Citations (document title, section, page).

    *   Optionally: confidence score, latency, token usage.

Key properties of the online pipeline

It must be fast.

Users expect answers in < 1 second, ideally < 500ms.
It must be stable under load.
- Handle many concurrent users.
- AvoidLatency spikes.
- Use caching, batching, and load balancing.
It must be observable.
- Log queries, retrieval results, latency.
- Track errors and failures.
- Monitor metrics (latency, cost, quality).

This pipeline is where you decide:

How queries are interpreted.
How retrieval is constrained.
How context is prepared for the LLM.
How answers are presented and tracked.

Why You Must Design Both Pipelines

Most tutorials only design the online pipeline:

Query
  ↓
Vector Search
  ↓
LLM
  ↓
Answer

They assume:

Documents are already chunked.
Chunks are already embedded.
Indices already exist.
Metadata is perfect.

In production, that assumption is fatal.

Example: A broken offline pipeline

Company A:

Uploads a PDF policy.
Extracts raw text.
Splits every 500 tokens.
Embeds.
Stores.

Company B:

Uploads the same PDF.
Detects structure: headings, sections, tables.
Cleans headers/footers, page numbers.
Creates section-based chunks.
Extracts metadata: document_type, version, language.
Embeds with domain-specific model.
Stores in vector + BM25 indexes.

Six months later:

Company A's RAG returns fragmented, noisy chunks.
Company B's RAG returns structured, filtered, relevant chunks.

The difference is not the LLM.

It's the offline pipeline.

Example: A broken online pipeline

Company C:

Uses only vector search.
No metadata filters.
No reranking.
Sends top-50 chunks to LLM.

Company D:

Uses hybrid search (vector + BM25).
Applies metadata filters (language, version, department).
Reranks with cross-encoder.
Sends top-5 compressed chunks to LLM.

Under the same LLM, Company D's answers are:

More precise.
More grounded.
More consistent.

The difference is not the LLM.

It's the online pipeline.

How to Think About These Pipelines

Treat them as two independent systems:

Offline = "data engineering" system.
Online = "search engine" system.

You can:

Change chunking strategies without touching retrieval logic.
Add new metadata fields without changing the LLM.
Swap embedding models without rewriting the query pipeline.
Introduce reranking without changing ingestion.

But you must design both.

If one is weak, the whole system fails.

In the next articles, we'll go deeper into each component of a production RAG system:

Part 2: Building a Production RAG Pipeline

We will explore:

Ingestion pipelines (parsers, cleaners, and document processing).
Chunking strategies with practical examples.
Metadata design and how it improves retrieval quality.

Part 3: Advanced Retrieval for Production RAG

We will explore:

Embeddings and semantic representation.
Hybrid search combining vector search and keyword retrieval.
Reranking strategies.
Context compression techniques.

Part 4: Scaling RAG Systems in Production

We will explore:

Prompt construction patterns.
Large-scale RAG architecture.
Performance optimization.
Scaling pipelines for millions of documents.

Part 5: Evaluating and Improving Production RAG Systems

We will explore:

RAG evaluation methods.
Production monitoring.
Common failure patterns.
15 production mistakes with real-world scenarios.
A complete production RAG checklist.

Together, these articles cover the complete journey from a simple RAG prototype to a reliable production-grade AI system.

Stop Starting with Microservices (Build a Monolith First)

Damir Karimov — Thu, 16 Jul 2026 11:07:51 +0000

Most startups don't fail because they chose a monolith.

They fail because they never built something people actually want.

The "Microservices from Day One" Trap

If you hang out on YouTube, Reddit, or X, you've probably seen this take:

"Serious applications use microservices."

It sounds right.

It sounds modern.

It sounds like what Netflix, Uber, and Amazon are doing.

So teams copy the architecture… before they even have 100 users.

What they end up with:

Weeks (or months) of infrastructure work
Multiple repositories
Complex CI/CD pipelines
Painful debugging across services
And barely any actual product

The real mistake isn't choosing a monolith.

It's solving problems you don't have yet.

Why Microservices Feel So Attractive

Let's be honest — microservices are fun.

You get to play with:

Docker
Kubernetes
API Gateways
Message brokers (Kafka, RabbitMQ)
Distributed tracing
Event-driven systems

It feels like real engineering.

But here's the uncomfortable truth:

Complexity does not create business value.

Your users don't care how your services communicate.

They care if your product works.

The Hidden Cost Nobody Mentions

Imagine you're building a SaaS.

Instead of one app, you split it into:

Auth service
User service
Billing service
Notification service
Analytics service

Looks clean on a diagram.

Now a user can't register.

Where's the bug?

Auth service?
User service?
API Gateway?
Kafka?
Network?
Database?
Docker config?
Kubernetes?

A simple feature now spans multiple services and repos.

Instead of building features…

you're chasing logs across your system.

What You Actually Get with a Monolith

A monolith isn't outdated.

It's just one deployable application.

That gives you:

One codebase
One deployment
One database
One place to debug
One place to search

For early-stage products, this means:

Faster development
Simpler debugging
Easier onboarding
Lower infrastructure costs
Fewer production surprises

Most importantly: you spend time building product, not infrastructure.

"But What About Scaling?"

This is the default objection.

Here's the reality:

Most apps never reach the point where a monolith is the bottleneck.

Examples:

Instagram started as a monolith
GitHub ran as a monolith for years
Shopify is still largely monolithic internally

Even at scale, many companies use modular monoliths, not pure microservices.

Scaling users matters more than scaling architecture.

The Real Problem: Bad Design

People often blame monoliths for what is actually poor code structure.

A bad monolith looks like:

Massive classes
Circular dependencies
Duplicated logic
No clear boundaries

But microservices don't fix bad design.

They distribute it.

Now instead of one messy codebase…

you have ten.

The Sweet Spot: Modular Monolith

You don't have to choose between chaos and complexity.

Start with a modular monolith.

Structure your app by domain:

src/
  auth/
  users/
  billing/
  notifications/
  analytics/
  shared/

Each module owns:

Business logic
Data access
Services
API layer

Modules talk through clear interfaces, not internal hacks.

You keep:

Simple deployment
Clean boundaries
Flexibility for future scaling

And when a module actually needs to be extracted…

you already know where to cut.

When Microservices Actually Make Sense

Microservices do solve real problems.

But only when those problems exist.

They make sense when:

Multiple teams need independent deployments
Different parts of the system scale differently
You need different tech stacks per service
Deployments are slowing you down
Failures must be isolated

Notice something?

These are not early-stage startup problems.

Architecture Is a Business Decision

Architecture is not a flex.

It's a trade-off.

Move to microservices when your business needs it — not when Twitter tells you it's "industry standard."

If deploy takes 30 seconds → you don't need Kubernetes
If one DB handles your load → don't split it into 10
If your team fits in one room → you don't need 20 services

A Simple Rule

Start simple.

Measure.

Find bottlenecks.

Then optimize.

Not the other way around.

Final Thought

The best architecture is not the most impressive one.

It's the one that helps your team ship valuable features every week.

Build the product first.

Earn the complexity later.

Because users don't buy architectures.

They buy solutions.

Why MCP Is Becoming the Standard Layer for AI Integrations (And Why Your Team Will Eventually Use It)

Damir Karimov — Wed, 08 Jul 2026 10:13:58 +0000

AI doesn't need another model.

It needs a sane, reusable way to connect models to the real world: tools, data, APIs, and workflows.

That's exactly the problem the Model Context Protocol (MCP) is solving.

Right now, every AI product team is quietly rebuilding the same thing:

a GitHub wrapper
a Jira wrapper
a Slack client
a DB adapter
a Drive integration

The model is fine.

The integration layer is chaos.

And that chaos is exactly why MCP is becoming the standard layer for AI integrations.

If your team is building AI products, this is not a “maybe later” problem. It's already your problem.

The real bottleneck in AI products isn’t the model

We love talking about reasoning, context windows, benchmarks.

But in real products, the hardest part is not intelligence — it's glue code.

Try building an assistant that:

creates a GitHub issue
checks Jira tickets
reads Slack threads
summarizes deployment logs

The challenge isn't the model. It's wiring four or five systems together, each with its own:

auth model
API shape
response format

So teams do the same thing again and again:

write custom connectors
duplicate logic
maintain brittle orchestration

You're not fighting the model.

You're fighting the integration layer.

MCP is not about making the model smarter.

It's about making the integration path repeatable, reusable, and standardized.

That's the game-changer.

What MCP actually is (without the hype)

MCP is an open protocol that standardizes how AI hosts connect to external capabilities.

Not in a vague future sense — but in a concrete architecture:

Host — your AI app (IDE, desktop assistant, etc.)
Server — exposes capabilities
Client layer — connects them via MCP

Instead of this:

"Integrate every tool separately for every AI client"

You get this:

"Expose capabilities once via a standard interface"

Important nuance:

MCP does not replace APIs.

It standardizes how AI systems discover and use them.

That's the core reason MCP is becoming the standard integration layer for AI.

Not because it's flashy.

But because it removes the most expensive, repetitive work from your team's plate.

The missing abstraction: tools, resources, prompts

A lot of shallow commentary reduces MCP to "tool calling."

That's incomplete.

MCP is cleaner if you think in three primitives:

Tools → actions (create issue, run query, deploy service)
Resources → data (files, logs, documents, records)
Prompts → reusable workflows (incident triage, debugging flows)

That separation matters.

It turns "AI calling random endpoints" into a structured system.

And structured systems are the only systems that scale.

Example

GitHub MCP server → tool: create issue
Docs server → resource: read documentation
Support system → prompt: incident triage workflow

That's a much better mental model than:

"Just give the AI API access"

And it's exactly the kind of structure that makes MCP a standard layer, not just another integration pattern.

Why the HTTP analogy is useful (and where it breaks)

The HTTP analogy is useful — but only at a high level.

HTTP made the web scalable because:

clients didn't care how servers were implemented
they just spoke a shared protocol

MCP aims for something similar:

AI hosts don't care how tools are implemented
they just use a standard interface

But let's be precise:

HTTP standardizes communication between systems

MCP standardizes how AI hosts discover and use capabilities

MCP does not replace HTTP.

It complements it — by normalizing the capability layer above raw APIs.

That's why "MCP is the HTTP of AI" is catchy, but also imprecise.

The stronger claim is:

MCP is becoming the standard layer AI systems use to connect to the software around them.

And that claim is far more actionable for engineers.

What this unlocks (in practice)

Imagine you're inside your IDE and you ask:

"Find the failed deployment, inspect logs, and create a GitHub issue."

Without MCP:

custom integrations
brittle orchestration
duplicated logic

With MCP:

The host discovers available MCP servers
Finds a logging server + GitHub server
Calls a logging tool → fetch logs
Reads structured output
Calls GitHub tool → creates issue
Returns final result

Nothing magical happened to the model.

The system just became composable and reusable.

That's the essence of a standard layer.

And that's what teams will care about when they're under pressure to ship faster, with fewer bugs.

When MCP is actually worth it

MCP is not a silver bullet.

But it becomes incredibly powerful in exactly the scenarios where most AI teams struggle:

Strong fit:

internal developer platforms
AI-powered IDEs
enterprise workflows
repeated tool access
multi-system orchestration

Weak fit:

simple one-off integrations
single API calls
small scripts or prototypes

If you don't feel integration pain yet, MCP will feel like overengineering.

If you do — it starts to look like infrastructure.

And once you've built even one serious AI product, you will feel that pain.

That's when MCP moves from "maybe" to "necessary".

The part no one wants to say

MCP won't save you from:

bad APIs
broken auth
poor observability
flaky tools
versioning chaos

It standardizes access.

It does not guarantee quality.

That's not a bug of MCP.

It's just reality: protocols only remove coordination cost, not all engineering trade-offs.

But even with that caveat, MCP still reduces fragmentation — which is the core problem it's meant to solve.

And fragmentation is the part that kills velocity in AI teams.

So why MCP is becoming the standard layer

Because this is the first serious attempt to standardize the integration layer of AI systems.

Not models.

Not prompts.

Integrations.

If adoption grows, a pattern emerges:

applications expose capabilities through MCP servers
AI hosts discover those capabilities dynamically
developers stop writing one-off adapters for every client

That's exactly what standard layers do:

they make patterns familiar
they reduce duplicated work
they turn bespoke glue code into reusable infrastructure

And that's why MCP is becoming the standard layer for AI integrations.

Final take: why you should care

If your team is building AI products today, you're already paying the cost of fragmented integrations.

You're paying it in:

time spent on custom connectors
bugs in brittle orchestration
pain when you need to add a new tool
fear when you think about scaling to more systems

MCP is not a slogan.

It's a practical, concrete way to reduce that cost.

Not flashy.

Not magical.

But if adoption keeps growing, MCP could become one of the default ways AI systems connect to everything else around them.

And for anyone building real AI products, that's exactly the layer worth paying attention to — and betting on.

Because the future of AI isn't just more models.

It's more connected, more reusable, less fragmented integrations.

And MCP is the first serious attempt to make that future real.

Why Your Background Jobs Fail in Production

Damir Karimov — Mon, 29 Jun 2026 10:41:23 +0000

Most developers first meet background jobs through a deceptively simple model: put work in a queue, let a worker pick it up, and assume the job will be processed once. In production, that mental model breaks quickly because queue systems are designed for reliability under failure, not for “perfect” one-shot execution.

Need to send an email? Queue it. Generate a PDF? Queue it. Process images? Queue it.

That feels elegant at first. The request finishes fast, the slow work happens elsewhere, and everything looks stable.

Then production happens.

A worker crashes after sending the email.

A network hiccup prevents the ACK from reaching the broker.

A third-party API times out halfway through processing.

The same job gets retried and runs again.

Now you have duplicate emails, duplicate charges, or jobs that appear to vanish and then come back later. That is not an edge case. That is normal distributed-systems behavior.

The happy path is a lie

The classic flow looks clean:

User registration

→ Create user

→ Push job to queue

→ Worker processes job

→ Send email

That flow is useful for explaining the idea, but it hides the real failure modes. In practice, each step can fail independently: enqueue can succeed while downstream storage is unavailable, a worker can complete side effects and then crash before acknowledging the message, or a retry can happen because the system cannot tell whether the previous attempt fully finished.

A queue does not guarantee execution exactly once.

It coordinates work, and it does so in a failure-prone environment.

That distinction matters because it changes how you design the job handler, the data model, and the retry strategy.

At-least-once delivery

Most production queue systems follow at-least-once delivery. Amazon SQS explicitly documents that a message can be received again, and it recommends designing applications to be idempotent so repeated processing does not cause harm.

That means a job may run more than once.

For example:

Worker receives job

→ Sends email

→ Crashes before ACK

→ Queue retries job

→ Email is sent again

From the queue’s point of view, the job was never confirmed as complete. Retrying is the correct behavior. The problem is that the side effect already happened.

This is why “the queue handled it” is not the same as “the system handled it.”

Idempotency is required

Once duplicate delivery is possible, idempotency becomes mandatory. The handler should produce the same final result even if it runs multiple times with the same input.

Bad example:

account.balance += 100;

If this runs twice, the user gets credited twice.

Better:

async function processPaymentJob(job: {
  transactionId: string;
  userId: string;
  amount: number;
}) {
  const alreadyProcessed = await db.paymentEvents.findUnique({
    where: { transactionId: job.transactionId },
  });

  if (alreadyProcessed) return;

  await db.$transaction(async (tx) => {
    await tx.paymentEvents.create({
      data: {
        transactionId: job.transactionId,
        userId: job.userId,
        amount: job.amount,
      },
    });

    await tx.accounts.update({
      where: { userId: job.userId },
      data: { balance: { increment: job.amount } },
    });
  });
}

The important part is not just the findUnique check. You also need a database constraint on transactionId, plus a transactional write boundary, so two workers cannot race and apply the same effect twice. That combination is much closer to production reality than a simple in-memory guard.

A good rule: store a stable business key for every side effect, then make that key unique.

Exactly-once is a trap

Engineers often ask whether exactly-once processing is possible. In practice, the answer is usually “not in the way people imagine.”

A job may charge a customer successfully, then fail while persisting the result. Or the database write may succeed while the worker crashes before confirming the message. Either way, the system ends up with inconsistent state unless it is explicitly designed for idempotency and reconciliation.

What people call “exactly-once” in production is usually a mix of:

deduplication,
idempotent handlers,
atomic state transitions,
checkpointing,
and careful recovery logic.

The useful mental model is not “never run twice.”

It is “running twice must not break anything.”

Retries can make outages worse

Retries are necessary, but uncontrolled retries can amplify a problem.

Imagine 50,000 jobs depending on a third-party API. If that API slows down or goes offline and every worker retries immediately, you create a retry storm. Instead of letting the system recover, you add more pressure to the failing dependency.

A better pattern is exponential backoff with jitter:

Retry #1 after a short delay
Retry #2 after a longer delay
Retry #3 after an even longer delay
Add random jitter so retries do not synchronize

function getRetryDelay(attempt: number) {
  const base = Math.min(60_000 * 2 ** attempt, 3_600_000);
  const jitter = base * (0.5 + Math.random() * 0.5);
  return Math.floor(jitter);
}

Without jitter, thousands of jobs can wake up at the same time and hit the same dependency together. With jitter, the pressure spreads out, which is exactly what you want during recovery.

Dead letter queues matter early

Some jobs are never going to succeed.

Examples include:

invalid payloads,
missing records,
deleted resources,
permanent validation errors.

If a job fails for a non-transient reason, retrying it forever wastes capacity and makes the queue harder to reason about. That is what a dead letter queue is for: a place to send messages that have exceeded their retry budget or failed in a non-recoverable way.

A healthy production setup usually treats DLQ as a first-class path, not an afterthought. It gives you a clear place to inspect poison messages, track recurring failures, and keep the main queue clean.

If you do one thing after adding retries, make sure you also define where failed jobs go next.

Long jobs should be split up

A common mistake is to make one job do everything:

generate a report,
download assets,
resize images,
upload files,
update the database,
send a notification.

That looks convenient until step five fails and you have to decide whether to rerun the whole thing. The larger the job, the harder it is to retry safely and the more expensive each failure becomes.

A better shape is a chain of smaller units:

generate report,
process assets,
upload results,
notify user.

Smaller jobs are easier to retry, easier to observe, and easier to scale independently. They also reduce blast radius when something breaks.

Monitoring matters more than processing

Many teams spend a lot of time building queue workers and very little time watching them. That is a mistake.

At minimum, track:

queue length,
queue lag,
processing latency,
job duration,
failure rate,
retry volume,
DLQ growth.

These signals tell you whether workers are keeping up, whether a dependency is slowing the pipeline, and whether failures are starting to accumulate. Azure’s background-job guidance also emphasizes reliability and scaling as first-class concerns, not optional extras.

If you cannot see your queue health, you are debugging blind.

Backpressure is the silent killer

Backpressure is what happens when work arrives faster than you can process it.

If your system receives 100 jobs per second but can only complete 50, the backlog grows continuously. Eventually that leads to rising latency, memory pressure, and a degraded user experience.

Common ways to deal with it include:

rate limiting,
worker autoscaling,
priority queues,
load shedding,
rejecting or delaying non-critical work.

Ignoring backpressure does not remove it. It just makes the failure slower and more expensive.

What production systems actually look like

What tutorials often show:

Queue → Worker → Done

What production systems usually need:

App

→ Queue

→ Workers

→ Retry policy

→ Dead letter queue

→ Metrics

→ Alerting

→ Recovery process

The processing code is rarely the hardest part. The hard part is designing for failure, visibility, and recovery.

Production checklist

Before shipping a queue-based system, verify that:

The job is idempotent.
The same job can run multiple times safely.
Retry delays use exponential backoff.
Jitter is enabled.
A dead letter queue exists.
Metrics and alerts are in place.
Failure scenarios were tested intentionally.
Workers can scale horizontally.
Backpressure is handled explicitly.
Jobs are small enough to retry without fear.

If several of these are missing, the system is probably more fragile than it looks.

Final thought

Background jobs fail in production not because queues are bad, but because the happy-path mental model is incomplete.

Real systems deal with crashes, retries, duplicate delivery, slow dependencies, and poisoned messages. The goal is not to pretend those failures do not exist. The goal is to make them survivable.

Once you design for failure instead of assuming success, background jobs become much more trustworthy.

Designing a Real-Time Chat System at Scale

Damir Karimov — Tue, 23 Jun 2026 09:02:11 +0000

Most developers underestimate how hard messaging systems really are.

A basic chat demo with WebSockets is easy to build. A production-grade messaging platform like WhatsApp, Telegram, Discord, or Slack is a completely different engineering problem. The hard part is not rendering messages in the UI. The hard parts are keeping millions of persistent connections alive, delivering messages reliably, preserving ordering, handling offline sync, scaling group chats, surviving partial failures, and keeping latency low around the world.

In this post, we'll design a scalable real-time chat architecture and walk through the trade-offs behind modern messaging systems.

Why chat gets hard

At first glance, chat seems simple:

send a message
store it
show it to the other user

That works for a demo. It falls apart fast in production.

Once you add real users, the system has to handle:

persistent connections
reconnects
message ordering
offline delivery
read receipts
typing indicators
multi-device sync
large group chats
global latency
partial failures

This is where chat stops being a UI feature and becomes a distributed systems problem.

Requirements

Before designing the system, we need clear requirements.

Functional requirements

Our chat system should support:

One-to-one chats
Group chats
Real-time message delivery
Read receipts
Typing indicators
Push notifications
Media attachments
Message history
Multi-device synchronization
Online and offline presence

Non-functional requirements

The system must also provide:

Low latency
High availability
Horizontal scalability
Fault tolerance
Reliable delivery
Event ordering
Efficient storage
Global distribution

At small scale, these are manageable. At millions of users, every one of them becomes a distributed systems concern.

High-level architecture

A modern chat platform is usually event-driven and distributed.

Clients
   │
   ▼
Load Balancer
   │
   ▼
WebSocket Gateway Cluster
   │
   ├── Authentication Service
   ├── Presence Service
   ├── Chat Service
   ├── Notification Service
   └── Media Service
          │
          ▼
    Kafka / Redis Streams
          │
          ▼
     Storage Layer

Each component has a separate responsibility. That separation is what makes the system scalable and easier to evolve.

Why polling fails

Many beginner chat apps start with polling:

GET /messages every 2 seconds

This works for a demo. It fails badly at scale.

Polling creates:

massive request overhead
unnecessary database reads
increased latency
battery drain on mobile devices
poor real-time responsiveness

Now imagine 2 million connected users polling every 2 seconds. That becomes 1 million requests per second even when nobody is sending anything. Real chat systems avoid this by using persistent connections.

Why WebSockets became the default

WebSockets keep a bidirectional connection open between the client and the server.

That gives you:

near real-time communication
lower overhead
reduced latency
efficient server push
better mobile performance

The client connects once:

Client ───── persistent connection ───── Server

After that, both sides can exchange events instantly. That is the foundation of modern messaging systems.

The hidden WebSocket cost

WebSockets solve one problem and introduce another.

A persistent connection is not free. Every connected user consumes:

a TCP socket
memory buffers
heartbeat state
authentication context

At 5 million concurrent users, this is no longer a normal API problem. It becomes a connection management problem.

That is why serious chat systems build specialized gateway infrastructure.

WebSocket gateway layer

The gateway layer is responsible for:

maintaining persistent connections
authenticating users
routing events
managing heartbeats
detecting disconnects

Gateways should stay as stateless as possible. Stateless gateways are much easier to scale, replace, and recover after failures.

Authentication flow

A common flow looks like this:

User logs in via HTTP.
Backend issues a JWT token.
Client opens a WebSocket connection.
Token is validated during the handshake.
Connection is associated with a user session.

Example:

wss://chat.example.com?token=JWT

After authentication, the gateway knows which user owns the connection.

Sending a message

Now let's look at the real delivery pipeline.

Suppose User A sends:

Hello

A simplified flow is:

Client sends a message event.
Gateway validates authentication.
Chat service validates permissions.
Message is persisted in durable storage.
Event is published into Kafka.
Recipient gateway receives the event.
Message is pushed to the recipient.
Delivery ACK is generated.
Read receipt is generated later.

This looks simple on paper. In reality, every step can fail.

Why persistence comes first

A lot of beginners try this:

deliver → save later

That is dangerous.

If the server crashes before persistence, the recipient saw the message, but the database lost it. Now the system is inconsistent.

Production systems usually do:

persist → publish → deliver

Durability comes first.

Message IDs and ordering

Distributed systems do not guarantee ordering automatically.

Example:

Message A
Message B

Recipient receives:

Message B
Message A

Why does this happen? Because messages may travel through different gateway servers, queues, and network paths.

Common ordering strategies

Timestamp ordering

Simple, but unreliable. Clock drift breaks consistency.

Incremental sequence IDs

More reliable.

Example:

conversation_id: 42

messages:
1
2
3
4

This guarantees ordering inside a conversation. Most real systems only guarantee local ordering per chat. Global ordering across the entire platform is usually impossible at scale.

Database design

Messaging systems are write-heavy. A popular group chat can generate thousands of writes per second.

Typical tables:

users
conversations
conversation_members
messages
message_status
attachments

The real challenge is partitioning.

Why a single database eventually fails

A single PostgreSQL instance works at the beginning. Over time, problems show up:

write bottlenecks
storage growth
replication lag
index size explosion

At scale, systems introduce sharding.

Sharding strategy

A common strategy is:

shard by conversation_id

Benefits:

messages for the same chat stay colocated
ordering becomes easier
queries stay efficient

Bad shard keys create hotspots. For example, sharding by user_id can spread large group chats across multiple shards and make fan-out expensive.

Kafka and event streaming

Modern messaging systems are heavily event-driven.

Kafka is useful because it provides:

durable event logs
replayability
partitioned scalability
consumer groups

Instead of services calling each other directly:

Chat Service → Kafka → Consumers

Consumers may include:

delivery service
notification service
analytics
moderation
push notifications

This decouples the system and makes failures easier to isolate.

Presence system

Presence is deceptively expensive.

Tracking online, offline, typing, and last_seen for millions of users creates a huge amount of event traffic. That is why most systems isolate presence into a dedicated service.

Presence implementation

A common architecture is:

Gateway → Redis → Presence Service

Gateways periodically send heartbeats.

Example:

PING every 30 seconds

If heartbeats stop, the user is marked offline.

Redis works well here because presence data is ephemeral. Not everything belongs in a relational database.

Typing indicators

Typing indicators look trivial. They are not.

Problems include:

high event frequency
noisy updates
unnecessary fan-out

Most systems heavily throttle typing events.

Example:

User typing → emit once every 3 seconds

Without throttling, typing indicators can overload the infrastructure faster than messages.

Fan-out in group chats

Suppose a group contains 500,000 users.

One message may require 500,000 deliveries. This is called fan-out.

Large fan-out is one of the hardest messaging problems.

Fan-out strategies

Fan-out on write

The server precomputes deliveries immediately.

fast reads
expensive writes

Fan-out on read

Messages are stored once.

cheaper writes
more expensive reads

Different platforms choose different trade-offs.

Offline synchronization

Users disconnect constantly:

mobile app closed
network loss
airplane mode
battery saver

The system must synchronize missed events efficiently.

Typical approach:

fetch all events after last_sequence_id

Example:

last_seen_message = 10451

Server returns:

10452+

Incremental synchronization is critical. Full synchronization is too expensive.

Push notifications

Offline users still need notifications.

Pipeline:

message event
   ↓
notification service
   ↓
APNs / FCM
   ↓
mobile device

Push systems are eventually consistent. Notifications may arrive late, duplicated, or out of order. Clients need to tolerate that.

Multi-device sync

Modern users expect sync across:

phone
desktop
browser
tablet

Each device may keep its own session. The backend tracks:

user_id
device_id
connection_id
last_sync_state

Events are usually delivered independently per device.

Reliability guarantees

Messaging systems need clear delivery semantics.

At-most-once

Fastest. Messages may disappear.

At-least-once

Reliable. Duplicates are possible.

Exactly-once

Extremely expensive and difficult in distributed systems.

Most production systems use:

at-least-once + idempotency

That is the practical choice.

Why exactly-once is mostly marketing

Exactly-once delivery across distributed infrastructure is incredibly hard.

Network failures create ambiguity:

Did the recipient receive the message?
Did the ACK get lost?
Did the retry create a duplicate?

Sometimes the sender cannot know for sure.

That is why many systems rely on retries, deduplication, and idempotent consumers instead of true exactly-once guarantees.

Common failure scenarios

Real systems fail all the time.

Examples:

gateway crashes
Kafka partition unavailable
Redis outage
slow consumers
duplicate events
partial synchronization
network splits

Reliable systems assume failure is normal.

Scaling strategies

As traffic grows, teams usually add:

horizontal scaling for gateway nodes
partitioned Kafka topics
regional infrastructure
CDN for media
caching to reduce database pressure

The biggest bottleneck is usually not CPU.

At scale, bottlenecks are often:

network throughput
memory usage
hot partitions
connection limits
disk I/O
replication lag

Chat systems are infrastructure-heavy workloads.

What makes messaging hard

The frontend UI may look simple. Underneath is a distributed system balancing:

consistency
availability
latency
reliability
scalability

Every architectural decision introduces trade-offs. You can optimize latency, durability, throughput, or cost, but rarely all at once.

That is why messaging systems remain one of the most interesting areas in system design.

Practical takeaway

If you are building chat, do not think only in terms of sockets and message lists. Think in terms of delivery guarantees, storage strategy, ordering, recovery, and fan-out.

A chat product becomes serious very quickly. The moment you need offline sync, presence, multi-device support, and reliable delivery, you are no longer building a UI feature. You are building a distributed messaging platform.

CI/CD for Modern Applications: From Manual Deployments to Reliable Delivery Pipelines

Damir Karimov — Mon, 15 Jun 2026 10:20:05 +0000

Last year my team had a production incident because someone manually deployed without running tests. The fix took 47 minutes. That's when I realized: CI/CD isn't about automation — it's about controlling risk in software delivery.

In this article, you'll learn how to design reliable pipelines for frontend (React/Next.js) and backend systems, including:

Artifact strategies and reproducible builds
Testing pyramid (unit, integration, contract, e2e)
Safe database migrations
Production observability and DORA metrics

Plus: a production checklist you can use immediately.

Modern application development is constrained less by coding speed and more by delivery reliability. Teams can ship features quickly, but production stability breaks when deployments remain manual or partially automated. CI/CD solves this by turning software delivery into a deterministic, repeatable pipeline.

What CI/CD Actually Solves

CI/CD (Continuous Integration and Continuous Delivery/Deployment) is not a toolset — it is a system design approach for software delivery.

Core problems it addresses

Problem	Impact
Integration conflicts in branches	Merge hell, delayed releases
Unpredictable release cycles	Business uncertainty, missed SLAs
Manual deployment errors	Production incidents, data corruption
Environment drift	"Works in CI, fails in prod"
Slow rollback processes	Extended MTTR, higher user impact

Goal: Not "faster deploys", but "reliable, repeatable deploys".

CI: Continuous Integration as a Validation Layer

Continuous Integration ensures every change is validated in a controlled pipeline before merging.

A minimal CI pipeline includes:

Linting (code quality enforcement)
Unit tests
Type checking (TypeScript, etc.)
Build verification

Example flow:

feature branch → PR → CI pipeline → checks pass → merge to main

Key principle: Every commit must result in a deployable artifact after CI. Without this rule, CI becomes cosmetic.

CD: Continuous Delivery vs Continuous Deployment

These terms are often mixed but differ structurally:

Aspect	Continuous Delivery	Continuous Deployment
Build state	Always deployable	Always deployable
Deployment trigger	Manual (human approval)	Automatic (after CI success)
Release decision	Controlled (gates, approvals)	Fully automated
Best for	Regulated/large systems	Fast-moving products with strong test coverage

CD controls to include:

Approval gates
Feature flags
RBAC & audit trails
Environment-specific configuration (not baked into artifacts)

Typical CI/CD Architecture

A production-grade pipeline includes:

Code → CI → Build artifact → Store → Deploy → Monitor

Components

Layer	Examples
Source control	GitHub, GitLab
CI runner	GitHub Actions, GitLab CI, Jenkins
Build system	Docker, native builds
Artifact storage	Docker registry, S3, artifact repo
Deployment system	Kubernetes, Vercel, ECS
Monitoring	Logs, metrics (Prometheus), traces (Jaeger)

Pipeline Stages in Modern Apps

A realistic pipeline is layered validation, not linear "test → build → deploy".

1. Pre-merge checks (on PR)

ESLint / Prettier
TypeScript compilation
Unit tests
Target: < 10 minutes

2. Post-merge CI

Full test suite
Integration tests
Security scanning (SAST)
Target: < 30 minutes

3. Build stage

Docker image build
Dependency locking
Artifact versioning (semver + git SHA)

4. Deployment stage

Staging rollout
Smoke tests
Production deployment (manual or automatic)

5. Post-deployment validation

Health checks
Observability signals (logs, metrics, traces)

Pipeline Internals: Artifacts, Provenance, Reproducibility

Artifact Strategy

Store immutable artifacts: Docker images, frontend build bundles, npm tarballs
Tag by semver + git SHA, store digest
Deploy by digest (not tag) to ensure consistency

Reproducible Builds

Use Docker images with pinned base versions
Lock dependency versions (package-lock.json, pnpm-lock.yaml)
Deterministic builds (no random timestamps)

Caching & Parallelization

Cache node_modules / pnpm store
Parallelize test suites
Incremental builds (Next.js, TypeScript)

Testing Strategy: The Pyramid

Test Type	When to Run	Purpose
Unit tests	Pre-merge (PR)	Fast feedback on logic
Integration	Post-merge	API, DB, service interactions
Contract tests	Post-merge	Cross-service compatibility
E2e tests	Staging/Nightly	Full user flow validation
Performance	Nightly	Latency, throughput baselines

Flaky tests:

Measure flakiness rate (target < 1–2%)
Quarantine flaky tests (separate suite)
Retry vs fix: fix critical flakies, quarantine low-priority

CI/CD in Frontend-heavy Systems (React / Next.js)

Typical Pipeline

Install dependencies (cached)
Lint + TypeScript check
Unit tests (Jest / Vitest)
Build (Next.js build)
Static analysis (bundle size, unused deps)
Deploy (Vercel / Docker / CDN)

Key Optimizations

Cache node_modules / pnpm store
Incremental builds
Split preview deployments per PR

CI/CD for Scalable Backend Systems

Backend Pipeline Requirements

Database migration strategy
Backward compatibility checks
Blue-green or canary deployments
Queue compatibility (Kafka / RabbitMQ)

Deployment Models

Model	Use Case	Risk
Rolling	Simple services	Medium
Blue-green	Critical services, fast rollback	Low
Canary	Gradual traffic shift, metrics	Low

Safe Database Migrations in CI/CD

Migration Checklist

Add column nullable
Deploy service with backward-compatible code (read new column with fallback)
Run backfill job
Make column non-nullable
Deploy service using new column without fallback
Clean up old code/columns

Orchestration:

Migration step in pipeline
Feature flag to enable new behavior
Health gate: error rate < 0.5%, latency P95 stable

Security in CI/CD

Modern pipelines must include:

Dependency scanning (npm audit, Snyk)
Secret detection
Container vulnerability scanning
Signed artifacts (supply chain security)
SBOM (Software Bill of Materials)

CI/CD without security gates becomes an attack surface.

Observability as Final Stage of CI/CD

Delivery is incomplete without feedback loop:

Logs (structured logging)
Metrics (latency, error rate)
Traces (request flow)
Alerting (SLA violations)

Key Pipeline & Production Metrics (DORA)

Metric	Target
Lead time for changes	< 1 day
Deployment frequency	Daily or on-demand
Change failure rate	< 5%
MTTR (Mean Time to Restore)	< 1 hour
Flaky-test rate	< 1–2%

CI/CD ends where production behavior becomes visible.

Common Failure Points in CI/CD Systems

Most failures are due to design flaws, not tools:

Failure Point	Risk
No clear artifact strategy	Inconsistent production builds
Missing rollback strategy	Increased risk, extended MTTR
Overloaded pipelines	Slow feedback, reduced developer productivity
Lack of environment parity	"Works in CI, fails in prod"

Rollback should be: Version switch (traffic to previous artifact digest) — NOT manual hotfix process

Production Checklist

Before promoting to production:

✅ Artifact immutable and signed

✅ DB migration safe (nullable → backfill → non-nullable)

✅ Feature flag ready for gradual enable

✅ Health gates configured (error rate, latency)

✅ Monitoring dashboards updated (deploy events visible)

✅ Rollback plan tested (traffic switch < 5 min)

Conclusion

CI/CD is not automation of deployment. It is a system for controlling risk in software delivery.

A well-designed pipeline ensures:

Predictable releases
Consistent environments
Fast feedback loops
Safe iteration speed

Teams that implement CI/CD correctly shift from "deploying code" to "managing delivery systems".

Real-Time Notification Systems Are Harder Than Most Teams Expect

Damir Karimov — Tue, 09 Jun 2026 10:26:13 +0000

If you’ve ever thought, “It’s just a WebSocket event,” this article is for you.

Notification systems look simple on the surface, but in production they fail in annoying, expensive, and user-visible ways:

duplicate notifications
missing events
race conditions
delayed delivery
mobile disconnects
retry storms
ordering bugs
state drift across regions

The tricky part is not sending a message.

The tricky part is making sure the right user gets the right notification, in the right order, with enough reliability that the system can survive crashes, retries, and mobile networks.

The problem with “just send a WebSocket event”

A basic notification flow looks like this:

Backend → WebSocket Server → Client

That works in local dev. It even works for a while in production.

Then real traffic arrives, and the system suddenly has to handle:

reconnects
offline users
multiple devices
persistence
retries
fan-out
backpressure
push fallbacks
deduplication
ordering

At that point, your “WebSocket feature” has become a distributed messaging system.

And that is a very different problem.

Delivery semantics matter first

Before you design the system, decide what guarantees you need.

Semantics	Meaning	Use case
At-most-once	Messages may be lost, but won’t be duplicated	Low-priority updates
At-least-once	Messages won’t be lost, but may be duplicated	Payments, security alerts
Effectively-once	Duplicates are removed with dedupe logic	Critical product events

Most teams make a mistake here.

They start building transport first, then discover later that they actually needed:

idempotency keys
durable cursors
sequence numbers
replay support
acknowledgements

That is why notification systems become expensive: the real problem is not delivery, it is delivery semantics.

Why fan-out breaks systems

One event is easy.

One event to 10,000 users is not.

A single action can trigger:

feed updates
badge counter updates
push notifications
email digests
analytics events
moderation triggers

That creates:

queue amplification
retry cascades
hot partitions
uneven load
latency spikes

So the system stops being “send a notification” and becomes “shape traffic safely under failure.”

Why duplicates happen

Duplicates usually do not come from one single bug. They appear from the interaction of retries, crashes, and missing idempotency.

A common chain looks like this:

1. Message is written to Kafka
2. Consumer processes it
3. Consumer crashes before committing offset
4. Partition is reassigned
5. Another consumer reads the same message
6. User gets the notification twice

That is not random.

That is at-least-once delivery with missing deduplication.

The fix

Use:

idempotency keys
dedupe storage
sequence numbers
consumer-side protection before side effects

Ordering is harder than throughput

Most users don’t complain about 200ms of delay.

They absolutely notice this:

“Payment refunded” arrives before “Payment received”

That destroys trust immediately.

Global ordering is usually too expensive. In practice, teams often choose:

per-user ordering
per-conversation ordering
approximate ordering
causal consistency where needed

For most products, per-user ordering is the best balance.

Example: sequence numbers per user

class NotificationLog:
    def __init__(self, user_id):
        self.user_id = user_id
        self.sequence = 0

    def append(self, notification):
        self.sequence += 1
        record = {
            "user_id": self.user_id,
            "sequence": self.sequence,
            "notification": notification,
            "idempotency_key": f"{self.user_id}:{self.sequence}"
        }
        kafka.produce(topic="notifications", key=self.user_id, value=record)
        return record

This is a simple way to keep ordering stable inside a user shard.

Mobile makes everything worse

Desktop clients are relatively stable.

Mobile clients are not.

You have to deal with:

app backgrounding
battery optimization
network switching
silent disconnects
delayed push delivery
OS throttling

That’s why real systems often combine:

WebSockets for active sessions
APNs for iOS
FCM for Android
polling or pull fallback
local persistence
sync checkpoints

Important detail

APNs and FCM are not guaranteed single-delivery transport.

They can:

delay notifications
drop messages under pressure
coalesce updates
expire tokens
throttle traffic

So if the notification matters, the server still needs durable state.

A real incident example

At 3AM, an on-call engineer gets paged because one user received dozens of duplicate payment emails.

What happened?

the consumer crashed mid-batch
the offset was not committed
Kafka redelivered the same event
the email sender had no dedupe check
the user got spammed

That kind of issue is painful because it is not one bug.

It is a chain of small design decisions that only becomes visible under failure.

The practical fix

def send_notification(notification):
    idempotency_key = f"{notification.user_id}:{notification.sequence}"

    if redis.exists(f"dedupe:{idempotency_key}"):
        consumer.commit()
        return

    email_service.send(notification)
    redis.setex(f"dedupe:{idempotency_key}", 24 * 3600, "1")
    consumer.commit()

The important part is not the code style.

It is the fact that the system now assumes duplicates can happen and is built to survive them.

Observability is not optional

If you cannot observe the pipeline, you cannot debug it.

Useful signals include:

queue lag
retry count
delivery success rate
connection churn
consumer health
fan-out latency
push provider errors

The real question is not:

Did we send the event?

It is:

Can we prove the user received it?

Those are very different questions.

Metrics worth tracking

Metric	Good target
Delivery success rate	>99.5%
p99 delivery latency	<500ms
Consumer lag	low and stable
Retry rate	close to zero
Connection churn	predictable

What to do in production

A good notification system needs a runbook, not just code.

If retries spike:

Throttle producers.
Pause non-critical workers.
Increase retry backoff.
Check consumer lag.
Check push provider errors.
Rehydrate missed clients from durable state.
Replay safely with idempotency keys.

That is what makes the system operationally survivable.

The scaling path

A lot of teams go through the same evolution.

Startup

API → WebSocket Server → Client

Mid-scale

API → Kafka → Notification Workers → WebSocket Gateway Cluster → Client

High-scale

API → Multi-region event bus → regional workers → regional gateways → clients

At higher scale, the hardest problems are usually:

state distribution
per-user ordering
region routing
dedupe
offline recovery

Not CPU.

State.

What strong teams optimize for

Early teams optimize for:

speed of delivery

Strong teams optimize for:

correctness
recoverability
observability
graceful degradation
idempotency

That difference matters a lot in production.

A notification that is slightly late is usually acceptable.

A notification that is duplicated, lost, or out of order is not.

Testing the system

You should test the failure modes, not just the happy path.

Try:

dropping WebSocket connections mid-message
killing consumers during processing
simulating mobile sleep/wake cycles
forcing Kafka rebalances
replaying duplicate events
load testing fan-out spikes

If the system only works when nothing fails, it is not ready.

Final thought

Real-time notification systems look simple until scale, retries, mobile behavior, ordering, and distributed state show up.

Then they become one of the hardest backend problems in the product.

The goal is not just to send events.

The goal is to make sure the right user gets the right notification, with the right semantics, even when the system is under stress.

AI Wrappers Are Dying: Why Most AI Products Fail

Damir Karimov — Wed, 27 May 2026 12:01:10 +0000

In 2026, building an app on top of OpenAI or Anthropic is easier than ever. But wrappers are dying.

A polished UI and a few RAG pipelines can get you to launch. They will not get you lasting advantage.

OpenAI API is not a competitive moat.

Wrappers Are Dying

The first wave of AI startups was inevitable. Foundation models became powerful enough that developers could ship useful products without training models from scratch. The barrier to entry dropped dramatically.

The market filled up with wrappers.

That was not irrational. It was the fastest way to test demand and prove people would pay for AI-enabled outcomes. For many founders, a wrapper was the right starting point. It reduced time-to-market and let them focus on distribution.

But wrappers that worked for speed do not work for defensibility.

Why Wrappers Are Fragile

A wrapper around an LLM is a thin interface over someone else's intelligence.

When the underlying model improves, your product advantage shrinks. When a competitor copies your UX, your edge disappears. When the model provider ships your core feature natively, your differentiation collapses overnight.

The closer your product is to a generic interface over a foundation model, the easier it is to clone.

Three problems:

The UI is visible and easy to imitate.
The prompts and workflows are often not deeply proprietary.
The core model capability is rented, not owned.

Many AI products compete on packaging rather than infrastructure.

If your product can be described as "ChatGPT, but for X," you have product-market fit risk before you have a moat.

What Creates Real Moat

A real moat in AI is not "we use GPT." It is owning something the next startup cannot easily replicate.

That includes:

Proprietary data
Embedded workflows
Deep enterprise integration
Distribution advantages
Domain-specific expertise
Feedback loops that improve the product over time

Model access is replaceable. Workflow capture is sticky.

If your product becomes part of how a team actually works, not just a tool they try once, you build defensibility. If you own the system of record, the approval flow, the compliance layer, or the operational pipeline, you are selling infrastructure, not AI.

The more your product learns from user behavior, customer data, and domain-specific outcomes, the harder it becomes to copy.

Moat Patterns That Survive

Proprietary Data Moat

If your product collects high-signal, domain-specific data that competitors cannot access, you improve faster over time.

Examples:

labeled support cases
medical annotations
legal review outcomes
sales conversation feedback
codebase-specific assistant traces

The moat works only if the data turns into better predictions, better retrieval, or better workflow decisions.

Workflow Moat

If your product becomes the place where work starts, gets reviewed, and gets approved, switching becomes painful.

Workflow moats require:

native integrations
permissions and access control
human-in-the-loop steps
audit logs
reliable outputs that fit existing processes

Enterprise AI products win by becoming infrastructure, not assistants.

Distribution Moat

If your product is embedded in Slack, email, CRM, IDEs, or internal tooling, it becomes harder to displace. Adoption is already inside the user's daily flow.

The best model in the world loses if users never reach it.

Trust and Compliance Moat

In regulated environments, trust is product value.

If you can prove data handling, retention rules, access controls, auditability, and predictable behavior, you compete on more than output quality. For enterprise buyers, this is the difference between a demo and a contract.

Cost and Infrastructure Moat

Some AI products create advantage by reducing inference cost, latency, or operational overhead at scale.

This moat is weaker than proprietary data or workflow lock-in. It matters when usage volume is high. If you deliver similar quality at lower cost, your margin improves and pricing flexibility increases.

RAG Alone Is Not Enough

RAG is useful. It is not a moat.

Retrieval connects foundation models to private corpora, internal docs, and customer-specific context. But if every competitor can index similar documents and call the same model, the architecture is not defensible.

RAG becomes valuable when paired with:

proprietary corpora
strong ranking and retrieval quality
feedback loops
domain-specific evaluation

The moat is not the retrieval layer. It is retrieval, data quality, and embedded usage over time.

Platform Dependency Is a Liability

The biggest hidden risk in AI startups is platform dependency.

If your roadmap depends on a single provider, you inherit their pricing, latency, policy changes, rate limits, and feature roadmap. That is not a moat. That is a liability.

When OpenAI improves a capability, it helps the whole market, including your competitors. When OpenAI ships a built-in feature that overlaps with your product, your differentiation evaporates overnight.

Relying entirely on external model APIs is dangerous for long-term architecture. The more your product is a front-end to a general model, the more exposed you are to commoditization.

Ask this: if model prices change, if output quality improves, or if the model vendor ships your core feature natively, what still makes you valuable?

Enterprise Workflows Are Where Winners Live

The strongest AI products solve a workflow that already exists inside a company. They do more than "answer questions."

Enterprise buyers care about more than output quality. They care about:

Access control
Compliance
Auditability
Data retention
Integrations with existing systems
Human approval steps
Reliability at scale

Workflow-based products have stronger moats than generic assistants. They do not just generate text. They become part of operational machinery.

Once AI is embedded in billing, support, procurement, legal review, or internal knowledge systems, switching costs rise quickly.

The best products feel "boring" from the outside. They are not flashy consumer apps. They are operational systems that save time, reduce risk, or increase throughput.

Vertical AI Wins

Vertical AI is stronger than horizontal AI because it combines domain data, workflow design, and distribution.

A vertical product knows the problem deeply. It understands terminology, edge cases, compliance rules, and customer expectations in a specific domain. This makes it harder to replace with a generic chatbot.

Proprietary data becomes especially important here. The more your product learns from a narrow, high-value domain, the more its quality ties to data that others do not have.

Winners connect three things:

domain-specific data
operational workflow
recurring business value

A good vertical AI product is deeply fitted to a single job. That fit becomes harder to copy with every interaction.

Which AI Companies Survive

AI companies that survive are not the ones with the flashiest demos. They turn model capability into durable product advantage.

They:

own proprietary or hard-to-access data
sit inside critical workflows
integrate deeply into enterprise systems
build operational infrastructure, not interfaces
create switching costs through usage, trust, and process

The model may be replaceable. The product around it should not be.

This is the difference between a temporary AI app and a lasting business.

How to Measure Moat

Signals that the moat is getting stronger:

Retention stays high even when model quality changes
Customers rely on the product as part of a repeatable workflow
The cost to replicate your dataset is high
More value comes from your proprietary layer than from the base model
Integrations increase switching costs over time
Unit economics improve as usage and feedback grow

Test: if a competitor copied your UI tomorrow, would they still need the same data, trust, integrations, and operational context to match your product?

If yes, you are building a real moat.

Conclusion

The problem with most AI products is not that they use AI. They confuse access to AI with defensibility.

A great interface gets attention. It rarely creates a moat. Real technical moats come from data, workflow, infrastructure, and integration — things hard to copy and harder to unwind.

The right question is not "How can we add a model?" The right question is: What do we own that becomes more valuable over time?

The best AI companies are not the ones with the loudest demo. They are the ones whose product gets more embedded, more trusted, and more expensive to replace every quarter.

Wrappers are dying. Build a moat instead.

Why Good Abstractions Make Debugging Harder

Damir Karimov — Thu, 21 May 2026 15:03:00 +0000

Good abstractions are great when you are building software.

They are much less great when you are debugging production.

The reason is simple: abstraction hides details, and debugging often depends on the details you hoped to ignore.

In small codebases, this is barely noticeable. In real systems, especially with caches, async flows, optimistic UI, and multiple state owners, it becomes a serious problem.

The core issue

The more layers you add, the easier it is for the system to become “locally correct” and “globally wrong”.

For example:

the frontend thinks the payment succeeded,
the backend committed the transaction,
the event was published,
the cache still serves the old value,
the UI shows stale data.

Every layer is doing something reasonable.

The problem is that they are not all talking about the same version of reality.

A simple example

Imagine this flow:

User clicks Retry payment
Frontend updates UI optimistically
API returns 200 OK
Database is updated
Event is sent to downstream systems
Redis still serves old state
UI refreshes from cache and shows stale data

This is the kind of bug that wastes hours.

Not because any single line of code is hard, but because the truth is spread across several places.

Example in code

Let’s say the frontend uses optimistic updates:

const onRetryPayment = async () => {
  setPaymentStatus("PAID");

  try {
    const response = await fetch("/api/payments/retry", {
      method: "POST",
    });

    if (!response.ok) {
      throw new Error("Retry failed");
    }
  } catch (error) {
    setPaymentStatus("FAILED");
  }
};

At first glance, this looks fine.

But now imagine:

the API succeeds,
the DB is updated,
an event is emitted,
a consumer deduplicates the event incorrectly,
Redis still contains the old value,
the UI re-renders from stale cache.

The bug is no longer in this function.

The bug is in the propagation path.

Why abstractions make this worse

Abstractions hide the exact mechanics that matter during incidents.

They hide things like:

who owns the state,
when the state changes,
whether the update is synchronous or async,
whether caches are invalidated,
whether retries are safe,
whether events can arrive out of order.

That is useful in normal development.

It is terrible during debugging.

Because when something is wrong, you do not need another clean interface. You need visibility.

Typical failure patterns

These are the patterns I see most often in real systems.

1. Stale read

The data was updated, but one layer still serves an old version.

// DB updated successfully
await db.payment.update({
  where: { id: paymentId },
  data: { status: "PAID" },
});

// Cache not invalidated

Result:

DB = PAID
cache = PENDING
UI = PENDING

2. Lost update

Two writes happen close together, and one silently overwrites the other.

await updateProfile({ name: "Alex" });
await updateProfile({ name: "John" });

If the system uses last-write-wins without proper locking or versioning, the final state may not match user intent.

3. Ghost update

One layer changes, but another never receives the update.

dispatch(updateOrderStatus("PAID"));
// but query cache is never invalidated

The result is a UI that looks stuck even though the backend is correct.

4. Event reorder bug

Events arrive in a different order than they were produced.

// Event B processed before Event A
processEvent("payment_succeeded");
processEvent("payment_pending");

Now the final state may be wrong even if both handlers are valid.

The debugging trap

The trap is assuming this is a code bug.

Very often it is not.

It is a state ownership bug.

That means the real question is not:

“Which function crashed?”

The real question is:

“Which layer is the source of truth right now?”

If you cannot answer that clearly, debugging becomes guesswork.

A better way to think about it

Instead of thinking in terms of “where is the bug?”, think in terms of “where does state live?”

A useful checklist:

Where is the canonical value stored?
Which layer may cache it?
Which layer may derive it?
Which layer may overwrite it?
Which layer may delay it?
Which layer may retry it?

If the same value exists in five places, you now have five opportunities for disagreement.

Debugging strategy

When a bug crosses abstraction boundaries, I usually inspect it in this order:

Step 1: Check the source of truth

Confirm where the canonical data lives.

Step 2: Rebuild the timeline

Trace the state from user action to backend write to cache update to UI read.

Step 3: Check invalidation

If a cache exists, verify it is updated or cleared at the right moment.

Step 4: Check idempotency

If retries or events are involved, verify the operation can safely happen more than once.

Step 5: Check ordering

If events are async, verify the system does not depend on strict ordering unless it actually guarantees it.

When abstractions do help

This is not an anti-abstraction argument.

Good abstractions are still valuable when they:

reduce search space,
make ownership clear,
keep state local,
expose transitions explicitly.

For example, a small component with local state is easier to debug than three caches and two event consumers trying to keep the same value in sync.

function Counter() {
  const [count, setCount] = useState(0);

  return (
    <button onClick={() => setCount(count + 1)}>
      Count: {count}
    </button>
  );
}

This is easy to reason about because there is one owner of the state.

That is the difference.

What to do in real systems

If you want abstractions to stay helpful in production, make them observable.

That means:

add logs at boundaries,
use trace IDs,
keep ownership explicit,
invalidate caches intentionally,
design retries to be safe,
avoid hidden duplicated state.

A good abstraction should reduce complexity, not hide the mechanics that make incidents debuggable.

Final thought

The best abstractions are honest.

They do not pretend the system is simpler than it is. They make the system easier to understand without hiding where truth lives.

That is why debugging gets harder as systems grow: not because abstraction is bad, but because abstraction is often too successful at hiding the exact thing you need under pressure.

AI-generated code doesn't fail loudly. It fails correctly-looking.

Damir Karimov — Wed, 13 May 2026 12:39:58 +0000

AI-generated code rarely breaks in obvious ways. It passes review, ships
to production, and behaves correctly in controlled scenarios. The
problem is what happens after: failures appear only under timing, load,
retries, or inconsistent state transitions.

The core issue is not obvious bugs. It is code that looks structurally
correct while silently ignoring real-world failure modes.

Why AI code feels correct

AI tends to generate implementations with strong surface-level signals:

consistent TypeScript types
standard architectural patterns
clean async/await flows
readable naming conventions
familiar framework usage

This produces a strong cognitive bias during review. The code does not
look "risky", so it is assumed to be correct.

The gap appears because readability is not equivalent to correctness
under production conditions.

Where AI-generated code typically fails

1. Concurrency and race conditions

async function updateProfile(data: Profile) {
  setLoading(true);

  const response = await api.updateProfile(data);

  setUser(response.user);
  setLoading(false);
}

This assumes a single linear execution.

In real systems:

multiple requests can run in parallel
responses can resolve out of order
later responses can overwrite newer state

Result: stale state overwrite without errors or crashes.

2. Optimistic updates without consistency guarantees

setTodos((prev) => [...prev, newTodo]);
await api.createTodo(newTodo);

This assumes success.

Failure scenarios:

request fails but UI is not rolled back
retry creates duplicate entries
frontend state diverges from backend state

The system remains "visually correct" while data integrity is broken.

3. Stale closures and lifecycle assumptions

useEffect(() => {
  const interval = setInterval(() => {
    console.log(count);
  }, 1000);

  return () => clearInterval(interval);
}, []);

This pattern locks in initial state.

In production:

values become stale over time
UI desynchronization occurs
behavior depends on render timing rather than logic

No runtime error occurs, so the issue is often missed.

4. Weak caching and invalidation logic

const cacheKey = `user-${userId}`;

if (cache[cacheKey]) return cache[cacheKey];

const data = await fetchUser(userId);
cache[cacheKey] = data;
return data;

This assumes:

stable data shape
stable identity rules
single write path

In real systems:

partial updates invalidate assumptions
multiple services mutate the same entity
cache becomes silently stale rather than obviously wrong

5. Hidden assumptions about APIs

AI can introduce plausible but non-existent APIs:

await auth.refreshSession({ force: true });
storage.invalidateAllQueries();

These patterns often:

look consistent with ecosystem conventions
pass code review without deep verification
fail only at runtime

This shifts errors from compile-time to production-time.

6. Accumulated lifecycle leaks

useEffect(() => {
  const controller = new AbortController();

  fetchData(controller.signal);

  return () => controller.abort();
}, [id]);

Individually correct, but when repeated across systems:

inconsistent cleanup patterns accumulate
aborted requests still resolve in edge cases
memory usage grows gradually
behavior becomes harder to reproduce

Systemic issue: reduced verification depth

The main shift introduced by AI-generated code is not implementation
speed, but review behavior.

Before AI, writing code required reasoning during implementation. After
AI:

code already looks complete
structure appears correct by default
reviewers focus on surface validation

This creates a subtle degradation in engineering discipline:

fewer edge-case simulations
less reasoning about concurrency
weaker validation of failure states
acceptance of "looks correct" as correctness

Impact on real systems

1. Frontend state drift

UI remains stable visually while backend state diverges.

2. Authentication and session issues

race conditions during token refresh
inconsistent logout handling
background requests using invalid sessions

3. Payments and idempotency problems

duplicate transactions
retries without deduplication
partial failure inconsistencies

4. Distributed system inconsistencies

assumption of ordering guarantees
reliance on immediate consistency
incorrect retry semantics

These issues are not immediately visible. They surface as rare,
non-reproducible incidents.

The real risk

AI does not generate obviously wrong code.

It generates code that satisfies:

type safety
structural conventions
expected patterns
readable abstractions

This creates false confidence during review.

The critical failure is not bugs themselves, but reduced skepticism
toward code that appears correct.

Once that happens, correctness is no longer actively verified. It is
assumed.

Conclusion

AI increases development speed, but it also changes how correctness is
perceived.

The danger is code that looks correct enough that nobody questions it deeply.

When that happens, production issues stop being introduced by obvious mistakes and start emerging from unexamined assumptions embedded in clean-looking code.