DEV Community: Damir Karimov

Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search

Damir Karimov — Thu, 23 Jul 2026 12:12:04 +0000

Building a reliable RAG system is not about connecting an LLM to a vector database. It requires a complete architecture covering data ingestion, retrieval, ranking, evaluation, and production operations.

This article is the first part of a five-part series where we will explore how modern RAG systems are designed, why naive implementations fail, and what it takes to build AI search systems that work reliably in production.

Production RAG Architecture Series

Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search (you are here)
Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design
Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking
Scaling RAG Systems: Production Architecture and Performance Optimization
Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns

Chapter 1 — Why Most RAG Systems Fail in Production

Demo works. Production fails.

Building a Retrieval-Augmented Generation system isn't hard.

Building one that people can trust is.

You've probably seen hundreds of tutorials:

Documents → Split into Chunks → Generate Embeddings → Store in Vector DB → Similarity Search → Send Context to LLM → Answer

It looks beautifully simple.

You can build a working chatbot in 30 minutes with LangChain, LlamaIndex, or Haystack.

Upload a PDF.

Generate embeddings.

Store them.

Ask questions.

Done.

For a demo, that's enough.

For production, that's the beginning of your problems.

Imagine this.

You built an internal AI assistant for your company's documentation.

Demo:

User: "How do I reset my password?"

Assistant: "Go to Settings → Security → Reset Password. You'll receive an email with a link."

Perfect. Everyone applauds. Project approved.

Two weeks later, support starts using it.

User: "Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"

Assistant: "Yes."

Actual answer from the policy document:

"Active invoices must be closed before upgrading from Professional to Enterprise."

Nobody notices immediately.

Support gives wrong advice.

Customer tries to upgrade.

Billing rejects.

Support spends hours resolving.

Management asks:

"I thought AI was supposed to reduce support workload."

Nothing crashed.

Embeddings were correct.

Vector DB was running.

Similarity search returned relevant chunks.

LLM generated fluent text.

Every component worked.

Yet the system failed.

That is what makes production RAG difficult.

Failures rarely come from broken software.

They come from broken architecture.

The tutorial pipeline vs the real pipeline

Most tutorials focus on this:

Query
  ↓
Vector Search
  ↓
LLM
  ↓
Answer

Real production systems look more like this:

Ingestion (Offline)

  Upload
    ↓
  Object Storage (S3 / GCS)
    ↓
  OCR / Parser (PDF, HTML, Markdown)
    ↓
  Content Cleaning (headers, footers, page numbers)
    ↓
  Chunking (semantic + structure-aware)
    ↓
  Metadata Extraction (doc type, language, version, section)
    ↓
  Embedding Workers
    ↓
    ├─→ Vector DB
    └─→ BM25 / Keyword Index

----------------------------

Retrieval (Online)

  User Query
    ↓
  Query Rewriting / Expansion
    ↓
  Metadata Filters (language, department, version)
    ↓
  Hybrid Search (Vector + BM25)
    ↓
  Reranking (Cross-Encoder)
    ↓
  Context Compression
    ↓
  Prompt Builder
    ↓
  LLM
    ↓
  Answer + Citations

The language model is often the smallest part of the architecture.

But it receives almost all attention.

Retrieval is not solved

The biggest misconception:

"Retrieval is a solved problem. Just use a vector DB."

It isn't.

Think about the Library of Congress.

Naive approach:

Throw all books into a giant pile.
Ask someone to find the right one by reading random pages.
Eventually, they might find it.

This is basically what many beginner RAG systems do:

Split everything into fixed-size chunks.
Embed.
Retrieve top-k by cosine similarity.
Feed to LLM.

Professional librarian:

Before searching, they narrow it down:

Language?
Publication year?
Author?
Category?
Edition?
Print or digital?
Academic or popular?

Only after reducing millions of possibilities to a few hundred, they start semantic search.

Professional retrieval works the same way.

Goal is not to search better.

Goal is to search less.

Vector DB is an index, not a database

Another common mistake:

"Vector DB replaces traditional databases."

No.

A vector DB is not your source of truth.
Think about Google:
- Google doesn't own the internet.
- It indexes the internet.
- If Google servers disappear, websites still exist.

Same with RAG:

Your documents should live somewhere else:
- PostgreSQL
- S3 / GCS / Azure Blob
- SharePoint / Confluence / Notion
Vector DB only stores an efficient representation for fast retrieval.
Delete your vectors → you can rebuild them.
Delete your original documents → you lost everything.

Many production systems fail because developers treat Vector DB as permanent storage.

Retrieval starts at ingestion

Another mistake:

"Retrieval begins when the user submits a question."

No.

Retrieval begins the moment a document enters your system.

Imagine two companies.

Company A:

Employee uploads PDF.
System extracts plain text.
Splits every 500 tokens.
Generates embeddings.
Stores them.
Done.

Company B:

Same PDF arrives.

Before embeddings, system asks:

Is this real text or scanned images? (OCR needed?)
Does it contain tables?
Headers / footers?
Multiple languages?
Version numbers?
Duplicate pages?
References to other documents?
Sensitive information?
Structured sections (chapters, clauses)?

Only after understanding the document, indexing begins.

Which system produces better answers 6 months later?

Not the one with the better LLM.

The one with better ingestion.

RAG is an information retrieval problem

If there's one idea to remember:

RAG is not an AI problem.

RAG is an information retrieval problem that happens to use AI.

That changes everything.

Instead of asking:

"Which LLM should I use?"

You start asking:

How should documents be represented?
How should information be organized?
What makes one chunk better than another?
Which metadata should be extracted?
Which ranking strategy should be used?
How do we measure retrieval quality?
How do we know retrieval failed?

Those matter more than choosing GPT-5, Claude, or Gemini.

A brilliant LLM cannot generate knowledge it never received.

Garbage retrieval → garbage answers. Every time.

Chapter 2 — The Toy RAG (with real code and examples)

Now let's see what a toy RAG really looks like, and why it fails.

We'll use Python + sentence-transformers + chromadb (local vector DB).

No LangChain magic, just raw code, so you see what's happening.

We'll build:

A naive RAG (fixed-size chunks).
Show where it breaks.
Compare with a slightly better version (structure-aware chunks + metadata).

Scenario: Company Knowledge Base

Imagine you have a simple knowledge base:

pricing_policy.md
onboarding_guide.md
support_rules.md

Example content from pricing_policy.md:

# Pricing Policy

## Upgrading Plans

Customers can upgrade from Professional to Enterprise.

**Important:** Active invoices must be closed before upgrading.

If you have any open invoices, contact billing@example.com.

## Downgrading Plans

Downgrading is allowed only if:
- No active trials
- No pending invoices

And a user query:

"Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"

Step 1: Naive RAG with fixed-size chunks

from pathlib import Path
from typing import List
import chromadb
from chromadb.config import Settings
from sentence_transformers import SentenceTransformer

# 1. Load documents
documents_dir = Path("docs")
texts = []
for md_file in documents_dir.glob("*.md"):
    texts.append(md_file.read_text(encoding="utf-8"))

# 2. Naive chunking: fixed size
def naive_chunk(text: str, chunk_size: int = 500, overlap: int = 50) -> List[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

all_chunks = []
for text in texts:
    all_chunks.extend(naive_chunk(text))

# 3. Embeddings
embedder = SentenceTransformer("all-MiniLM-L6-v2")
chunk_embeddings = embedder.encode(all_chunks, convert_numpy=True)

# 4. Store in Chroma
chroma_client = chromadb.Client(Settings(
    chroma_db_impl="duckdb+parquet",
    persist_directory="./chroma_db"
))
collection = chroma_client.create_collection("naive_rag")

for i, chunk in enumerate(all_chunks):
    collection.add(
        ids=[f"chunk_{i}"],
        embeddings=[chunk_embeddings[i].tolist()],
        documents=[chunk]
    )

# 5. Query
def query_naive_rag(collection, query: str, top_k: int = 5) -> List[str]:
    query_emb = embedder.encode([query], convert_numpy=True)[0].tolist()
    result = collection.query(
        query_embeddings=[query_emb],
        n_results=top_k
    )
    return result["documents"][0]

query = "Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"
retrieved = query_naive_rag(collection, query, top_k=5)

for i, chunk in enumerate(retrieved, 1):
    print(f"Chunk {i}:\n{chunk}\n")

What you'll likely see:

Some chunks might contain:

"Customers can upgrade from Professional to Enterprise."
"Downgrading is allowed only if..."
Random fragments with "invoices" but not the right rule.

But the crucial sentence:

"Active invoices must be closed before upgrading from Professional to Enterprise."

might be:

Split across two chunks.
Buried in a chunk with unrelated text.
Not in top-k at all.

Then LLM will generate:

"Yes, they can upgrade."

Because the retrieved context doesn't say "No".

This is exactly how naive RAG fails.

Not because embedding is wrong.

Because chunking + retrieval are not designed for real questions.

Step 2: Slightly better RAG with structure-aware chunks + metadata

Now let's do it more thoughtfully.

We'll:

Split by Markdown headings.
Keep section title + content together.
Add metadata: doc_name, section, language.

import re
from typing import List, Dict

def structure_aware_chunk(text: str, doc_name: str) -> List[Dict]:
    chunks = []

    # Split by headings: # Section, ## Subsection
    pattern = r"(^#{1,2}\s+.+$)"
    parts = re.split(pattern, text, flags=re.MULTILINE)

    # parts: [before_first_heading, heading1, content1, heading2, content2, ...]

    current_heading = None
    current_content = []

    for i, part in enumerate(parts):
        if i == 0 and part.strip() == "":
            continue
        if re.match(pattern, part):
            # New heading
            current_heading = part.strip()
            current_content = []
        else:
            # Content
            if current_heading:
                current_content.append(part.strip())

            if current_heading and (i + 1 >= len(parts) or re.match(pattern, parts[i + 1])):
                # End of section
                content_text = "\n\n".join(current_content).strip()
                if content_text:
                    chunks.append({
                        "text": f"{current_heading}\n\n{content_text}",
                        "metadata": {
                            "doc_name": doc_name,
                            "section": current_heading,
                            "language": "en",
                        }
                    })
                    current_heading = None
                    current_content = []

    return chunks

all_chunks = []
for md_file in documents_dir.glob("*.md"):
    doc_name = md_file.name
    text = md_file.read_text(encoding="utf-8")
    all_chunks.extend(structure_aware_chunk(text, doc_name))

# Embeddings + store (similar to before, but with metadata)
chunk_embeddings = embedder.encode([c["text"] for c in all_chunks], convert_numpy=True)

collection2 = chroma_client.create_collection("better_rag")

for i, chunk in enumerate(all_chunks):
    collection2.add(
        ids=[f"chunk_{i}"],
        embeddings=[chunk_embeddings[i].tolist()],
        documents=[chunk["text"]],
        metadatas=[chunk["metadata"]]
    )

def query_better_rag(collection, query: str, top_k: int = 5) -> List[Dict]:
    query_emb = embedder.encode([query], convert_numpy=True)[0].tolist()
    result = collection.query(
        query_embeddings=[query_emb],
        n_results=top_k
    )
    # Combine text + metadata
    return [
        {"text": doc, "metadata": meta}
        for doc, meta in zip(result["documents"][0], result["metadatas"][0])
    ]

retrieved2 = query_better_rag(collection2, query, top_k=5)

for i, item in enumerate(retrieved2, 1):
    print(f"Chunk {i} (from {item['metadata']['doc_name']}, section: {item['metadata']['section']}):")
    print(item["text"])
    print()

What improves:

Chunks now preserve sections:
- "## Upgrading Plans" + full content.
Metadata tells you:
- Which document.
- Which section.
Retrieval more likely to find the exact section about upgrading + invoices.

Now context to LLM might look like:

## Upgrading Plans

Customers can upgrade from Professional to Enterprise.

Important: Active invoices must be closed before upgrading.

If you have any open invoices, contact billing@example.com.

Then LLM can answer:

"No. Active invoices must be closed before upgrading from Professional to Enterprise."

Same embedding model.

Same LLM.

But better chunking + metadata → better retrieval → better answer.

What this shows

Even in this tiny example:

Fixed-size chunks → fragmentation → wrong answer.
Structure-aware chunks + metadata → better context → correct answer.

In production, differences are even bigger:

Millions of documents.
Many languages.
Many versions.
Complex queries.

If you don't design ingestion, chunking, and metadata properly, no LLM will save you.

Chapter 3 — Every Production RAG Has Two Pipelines

If you've only followed tutorials, you probably think RAG is one pipeline:

Query
  ↓
Retrieve
  ↓
Generate
  ↓
Answer

That's enough for a demo.

It's catastrophic for production.

In any real system, you actually have two separate pipelines:

Offline (ingestion) pipeline – processes documents.
Online (query) pipeline – handles user requests.

They have completely different goals, constraints, and failure modes.

Designing them as one monolithic flow is one of the first mistakes that leads to production failures.

The Offline Pipeline: Where Documents Live Their "Slow Life"

The offline pipeline is what happens to documents before anyone ever asks a question.

Offline Pipeline (ingestion)

  Documents (PDF, HTML, Markdown, JSON)
    ↓
  Parser (structure-aware)
    ↓
  Cleaner (headers, footers, page numbers)
    ↓
  Chunker (semantic / structure-aware)
    ↓
  Metadata Extractor
    ↓
  Embedder
    ↓
  Indexer
    ├─→ Vector DB
    └─→ BM25 / Keyword Index

----------------------------

Online Pipeline (query)

  User Query
    ↓
  Query Processor (rewrite, expand)
    ↓
  Metadata Filters
    ↓
  Hybrid Search (Vector + BM25)
    ↓
  Reranker (Cross-Encoder)
    ↓
  Context Compression
    ↓
  Prompt Builder
    ↓
  LLM
    ↓
  Answer + Citations

What this pipeline does

Parser

*   Reads raw documents.

*   Extracts structure: headings, tables, lists, code blocks, paragraphs.

*   For PDFs: uses OCR if needed, understands layout.

*   For Markdown / HTML: preserves semantic tags.

Cleaner

*   Removes noise:

    *   Headers / footers ("Company", "Confidential").

    *   Page numbers.

    *   Watermarks.

    *   Repeated boilerplate text.

*   Normalizes whitespace, encoding, line breaks.

*   Fixes broken sentences or artifacts.

Chunker

*   Splits documents into meaningful chunks.

*   Ensures:

    *   Headings are not cut in the middle.

    *   Tables stay intact.

    *   Paragraph boundaries are respected.

*   Uses strategies like:

    *   Structure-aware chunking

    *   Semantic chunking

    *   Parent-child chunking

    *   (We'll cover these in detail in Chapter 5.)

Metadata Extractor

*   Extracts document-level metadata:

    *   `document_type`

    *   `language`

    *   `version`

    *   `department`

    *   `created_at`, `updated_at`

*   Extracts chunk-level metadata:

    *   `section` / `heading`

    *   `page`

    *   `chapter`

    *   `parent_id`

    *   `hierarchy_path`

Embedder

*   Computes embeddings for each chunk.

*   Optionally computes document-level embeddings too.

*   Uses models appropriate for:

    *   Domain (technical, legal, general).

    *   Language (English, Russian, multilingual).

Indexer

*   Inserts chunks into:

    *   Vector DB (dense embeddings).

    *   BM25 / keyword index (sparse).

    *   (Optionally) graph index (for knowledge graphs).

Key properties of the offline pipeline

It doesn't need to be fast.

Processing a document can take seconds or minutes.

Documents are not requested by users in real time.
It must be correct and robust.

If ingestion is wrong, retrieval will never be right.

Bad chunks, missing metadata, wrong parsing → garbage retrieval.
It can be batched or incremental.
- Batch: nightly jobs, reprocess entire corpus.
- Incremental: process only new/updated documents.
- Event-driven: triggered by upload events via queues.

This pipeline is where you decide:

How documents are represented.
How information is organized.
What metadata is available for filtering.
What kind of chunks the retriever will see.

If you ignore this pipeline and only focus on "query → retrieve → generate", you're building a demo, not a production system.

The Online Pipeline: Where Requests Live Their "Fast Life"

The online pipeline is what happens when a user asks a question.

Online Pipeline (query)

  User Query
    ↓
  Query Processor (rewrite, expand, normalize)
    ↓
  Metadata Filters (language, department, version, etc.)
    ↓
  Hybrid Search (Vector + BM25)
    ↓
  Reranker (Cross-Encoder)
    ↓
  Context Compression
    ↓
  Prompt Builder
    ↓
  LLM
    ↓
  Answer + Citations

What this pipeline does

Query Processor

*   Rewrites or expands the query:

    *   "upgrade plan" → "upgrade from Professional to Enterprise plan".

    *   Adds implicit context (user role, department).

*   Normalizes:

    *   Lowercases, removes noise.

    *   Handles typos, slang, abbreviations.

Metadata Filters

*   Applies filters based on:

    *   User's language.

    *   User's department / team.

    *   Document version (latest, stable, deprecated).

    *   Document type (policy, doc, article).

*   Reduces the candidate set before search.

Hybrid Search

*   Runs:

    *   Vector search (semantic similarity).

    *   BM25 / keyword search (exact term matching).

*   Combines scores:

    *   `final_score = α * vector_score + β * bm25_score`.

Reranker

*   Takes top-K candidates (e.g., 50–100).

*   Uses a cross-encoder to compute a more precise relevance score for each.

*   Sorts and picks the best N (e.g., 5–10).

Context Compression

*   Limits the number of chunks sent to the LLM.

*   Drops low-relevance chunks.

*   Optionally summarizes or extracts key sentences.

*   Ensures the prompt stays within token limits.

Prompt Builder

*   Constructs the final prompt:

    *   System instructions.

    *   Retrieved context.

    *   User query.

    *   Output format instructions (citations, confidence, etc.).

LLM

*   Generates the answer based on the prompt.

*   Ideally grounded in the retrieved context.

Answer + Citations

*   Returns:

    *   The final answer.

    *   Citations (document title, section, page).

    *   Optionally: confidence score, latency, token usage.

Key properties of the online pipeline

It must be fast.

Users expect answers in < 1 second, ideally < 500ms.
It must be stable under load.
- Handle many concurrent users.
- AvoidLatency spikes.
- Use caching, batching, and load balancing.
It must be observable.
- Log queries, retrieval results, latency.
- Track errors and failures.
- Monitor metrics (latency, cost, quality).

This pipeline is where you decide:

How queries are interpreted.
How retrieval is constrained.
How context is prepared for the LLM.
How answers are presented and tracked.

Why You Must Design Both Pipelines

Most tutorials only design the online pipeline:

Query
  ↓
Vector Search
  ↓
LLM
  ↓
Answer

They assume:

Documents are already chunked.
Chunks are already embedded.
Indices already exist.
Metadata is perfect.

In production, that assumption is fatal.

Example: A broken offline pipeline

Company A:

Uploads a PDF policy.
Extracts raw text.
Splits every 500 tokens.
Embeds.
Stores.

Company B:

Uploads the same PDF.
Detects structure: headings, sections, tables.
Cleans headers/footers, page numbers.
Creates section-based chunks.
Extracts metadata: document_type, version, language.
Embeds with domain-specific model.
Stores in vector + BM25 indexes.

Six months later:

Company A's RAG returns fragmented, noisy chunks.
Company B's RAG returns structured, filtered, relevant chunks.

The difference is not the LLM.

It's the offline pipeline.

Example: A broken online pipeline

Company C:

Uses only vector search.
No metadata filters.
No reranking.
Sends top-50 chunks to LLM.

Company D:

Uses hybrid search (vector + BM25).
Applies metadata filters (language, version, department).
Reranks with cross-encoder.
Sends top-5 compressed chunks to LLM.

Under the same LLM, Company D's answers are:

More precise.
More grounded.
More consistent.

The difference is not the LLM.

It's the online pipeline.

How to Think About These Pipelines

Treat them as two independent systems:

Offline = "data engineering" system.
Online = "search engine" system.

You can:

Change chunking strategies without touching retrieval logic.
Add new metadata fields without changing the LLM.
Swap embedding models without rewriting the query pipeline.
Introduce reranking without changing ingestion.

But you must design both.

If one is weak, the whole system fails.

In the next articles, we'll go deeper into each component of a production RAG system:

Part 2: Building a Production RAG Pipeline

We will explore:

Ingestion pipelines (parsers, cleaners, and document processing).
Chunking strategies with practical examples.
Metadata design and how it improves retrieval quality.

Part 3: Advanced Retrieval for Production RAG

We will explore:

Embeddings and semantic representation.
Hybrid search combining vector search and keyword retrieval.
Reranking strategies.
Context compression techniques.

Part 4: Scaling RAG Systems in Production

We will explore:

Prompt construction patterns.
Large-scale RAG architecture.
Performance optimization.
Scaling pipelines for millions of documents.

Part 5: Evaluating and Improving Production RAG Systems

We will explore:

RAG evaluation methods.
Production monitoring.
Common failure patterns.
15 production mistakes with real-world scenarios.
A complete production RAG checklist.

Together, these articles cover the complete journey from a simple RAG prototype to a reliable production-grade AI system.

Stop Starting with Microservices (Build a Monolith First)

Damir Karimov — Thu, 16 Jul 2026 11:07:51 +0000

Most startups don't fail because they chose a monolith.

They fail because they never built something people actually want.

The "Microservices from Day One" Trap

If you hang out on YouTube, Reddit, or X, you've probably seen this take:

"Serious applications use microservices."

It sounds right.

It sounds modern.

It sounds like what Netflix, Uber, and Amazon are doing.

So teams copy the architecture… before they even have 100 users.

What they end up with:

Weeks (or months) of infrastructure work
Multiple repositories
Complex CI/CD pipelines
Painful debugging across services
And barely any actual product

The real mistake isn't choosing a monolith.

It's solving problems you don't have yet.

Why Microservices Feel So Attractive

Let's be honest — microservices are fun.

You get to play with:

Docker
Kubernetes
API Gateways
Message brokers (Kafka, RabbitMQ)
Distributed tracing
Event-driven systems

It feels like real engineering.

But here's the uncomfortable truth:

Complexity does not create business value.

Your users don't care how your services communicate.

They care if your product works.

The Hidden Cost Nobody Mentions

Imagine you're building a SaaS.

Instead of one app, you split it into:

Auth service
User service
Billing service
Notification service
Analytics service

Looks clean on a diagram.

Now a user can't register.

Where's the bug?

Auth service?
User service?
API Gateway?
Kafka?
Network?
Database?
Docker config?
Kubernetes?

A simple feature now spans multiple services and repos.

Instead of building features…

you're chasing logs across your system.

What You Actually Get with a Monolith

A monolith isn't outdated.

It's just one deployable application.

That gives you:

One codebase
One deployment
One database
One place to debug
One place to search

For early-stage products, this means:

Faster development
Simpler debugging
Easier onboarding
Lower infrastructure costs
Fewer production surprises

Most importantly: you spend time building product, not infrastructure.

"But What About Scaling?"

This is the default objection.

Here's the reality:

Most apps never reach the point where a monolith is the bottleneck.

Examples:

Instagram started as a monolith
GitHub ran as a monolith for years
Shopify is still largely monolithic internally

Even at scale, many companies use modular monoliths, not pure microservices.

Scaling users matters more than scaling architecture.

The Real Problem: Bad Design

People often blame monoliths for what is actually poor code structure.

A bad monolith looks like:

Massive classes
Circular dependencies
Duplicated logic
No clear boundaries

But microservices don't fix bad design.

They distribute it.

Now instead of one messy codebase…

you have ten.

The Sweet Spot: Modular Monolith

You don't have to choose between chaos and complexity.

Start with a modular monolith.

Structure your app by domain:

src/
  auth/
  users/
  billing/
  notifications/
  analytics/
  shared/

Each module owns:

Business logic
Data access
Services
API layer

Modules talk through clear interfaces, not internal hacks.

You keep:

Simple deployment
Clean boundaries
Flexibility for future scaling

And when a module actually needs to be extracted…

you already know where to cut.

When Microservices Actually Make Sense

Microservices do solve real problems.

But only when those problems exist.

They make sense when:

Multiple teams need independent deployments
Different parts of the system scale differently
You need different tech stacks per service
Deployments are slowing you down
Failures must be isolated

Notice something?

These are not early-stage startup problems.

Architecture Is a Business Decision

Architecture is not a flex.

It's a trade-off.

Move to microservices when your business needs it — not when Twitter tells you it's "industry standard."

If deploy takes 30 seconds → you don't need Kubernetes
If one DB handles your load → don't split it into 10
If your team fits in one room → you don't need 20 services

A Simple Rule

Start simple.

Measure.

Find bottlenecks.

Then optimize.

Not the other way around.

Final Thought

The best architecture is not the most impressive one.

It's the one that helps your team ship valuable features every week.

Build the product first.

Earn the complexity later.

Because users don't buy architectures.

They buy solutions.

Why MCP Is Becoming the Standard Layer for AI Integrations (And Why Your Team Will Eventually Use It)

Damir Karimov — Wed, 08 Jul 2026 10:13:58 +0000

AI doesn't need another model.

It needs a sane, reusable way to connect models to the real world: tools, data, APIs, and workflows.

That's exactly the problem the Model Context Protocol (MCP) is solving.

Right now, every AI product team is quietly rebuilding the same thing:

a GitHub wrapper
a Jira wrapper
a Slack client
a DB adapter
a Drive integration

The model is fine.

The integration layer is chaos.

And that chaos is exactly why MCP is becoming the standard layer for AI integrations.

If your team is building AI products, this is not a “maybe later” problem. It's already your problem.

The real bottleneck in AI products isn’t the model

We love talking about reasoning, context windows, benchmarks.

But in real products, the hardest part is not intelligence — it's glue code.

Try building an assistant that:

creates a GitHub issue
checks Jira tickets
reads Slack threads
summarizes deployment logs

The challenge isn't the model. It's wiring four or five systems together, each with its own:

auth model
API shape
response format

So teams do the same thing again and again:

write custom connectors
duplicate logic
maintain brittle orchestration

You're not fighting the model.

You're fighting the integration layer.

MCP is not about making the model smarter.

It's about making the integration path repeatable, reusable, and standardized.

That's the game-changer.

What MCP actually is (without the hype)

MCP is an open protocol that standardizes how AI hosts connect to external capabilities.

Not in a vague future sense — but in a concrete architecture:

Host — your AI app (IDE, desktop assistant, etc.)
Server — exposes capabilities
Client layer — connects them via MCP

Instead of this:

"Integrate every tool separately for every AI client"

You get this:

"Expose capabilities once via a standard interface"

Important nuance:

MCP does not replace APIs.

It standardizes how AI systems discover and use them.

That's the core reason MCP is becoming the standard integration layer for AI.

Not because it's flashy.

But because it removes the most expensive, repetitive work from your team's plate.

The missing abstraction: tools, resources, prompts

A lot of shallow commentary reduces MCP to "tool calling."

That's incomplete.

MCP is cleaner if you think in three primitives:

Tools → actions (create issue, run query, deploy service)
Resources → data (files, logs, documents, records)
Prompts → reusable workflows (incident triage, debugging flows)

That separation matters.

It turns "AI calling random endpoints" into a structured system.

And structured systems are the only systems that scale.

Example

GitHub MCP server → tool: create issue
Docs server → resource: read documentation
Support system → prompt: incident triage workflow

That's a much better mental model than:

"Just give the AI API access"

And it's exactly the kind of structure that makes MCP a standard layer, not just another integration pattern.

Why the HTTP analogy is useful (and where it breaks)

The HTTP analogy is useful — but only at a high level.

HTTP made the web scalable because:

clients didn't care how servers were implemented
they just spoke a shared protocol

MCP aims for something similar:

AI hosts don't care how tools are implemented
they just use a standard interface

But let's be precise:

HTTP standardizes communication between systems

MCP standardizes how AI hosts discover and use capabilities

MCP does not replace HTTP.

It complements it — by normalizing the capability layer above raw APIs.

That's why "MCP is the HTTP of AI" is catchy, but also imprecise.

The stronger claim is:

MCP is becoming the standard layer AI systems use to connect to the software around them.

And that claim is far more actionable for engineers.

What this unlocks (in practice)

Imagine you're inside your IDE and you ask:

"Find the failed deployment, inspect logs, and create a GitHub issue."

Without MCP:

custom integrations
brittle orchestration
duplicated logic

With MCP:

The host discovers available MCP servers
Finds a logging server + GitHub server
Calls a logging tool → fetch logs
Reads structured output
Calls GitHub tool → creates issue
Returns final result

Nothing magical happened to the model.

The system just became composable and reusable.

That's the essence of a standard layer.

And that's what teams will care about when they're under pressure to ship faster, with fewer bugs.

When MCP is actually worth it

MCP is not a silver bullet.

But it becomes incredibly powerful in exactly the scenarios where most AI teams struggle:

Strong fit:

internal developer platforms
AI-powered IDEs
enterprise workflows
repeated tool access
multi-system orchestration

Weak fit:

simple one-off integrations
single API calls
small scripts or prototypes

If you don't feel integration pain yet, MCP will feel like overengineering.

If you do — it starts to look like infrastructure.

And once you've built even one serious AI product, you will feel that pain.

That's when MCP moves from "maybe" to "necessary".

The part no one wants to say

MCP won't save you from:

bad APIs
broken auth
poor observability
flaky tools
versioning chaos

It standardizes access.

It does not guarantee quality.

That's not a bug of MCP.

It's just reality: protocols only remove coordination cost, not all engineering trade-offs.

But even with that caveat, MCP still reduces fragmentation — which is the core problem it's meant to solve.

And fragmentation is the part that kills velocity in AI teams.

So why MCP is becoming the standard layer

Because this is the first serious attempt to standardize the integration layer of AI systems.

Not models.

Not prompts.

Integrations.

If adoption grows, a pattern emerges:

applications expose capabilities through MCP servers
AI hosts discover those capabilities dynamically
developers stop writing one-off adapters for every client

That's exactly what standard layers do:

they make patterns familiar
they reduce duplicated work
they turn bespoke glue code into reusable infrastructure

And that's why MCP is becoming the standard layer for AI integrations.

Final take: why you should care

If your team is building AI products today, you're already paying the cost of fragmented integrations.

You're paying it in:

time spent on custom connectors
bugs in brittle orchestration
pain when you need to add a new tool
fear when you think about scaling to more systems

MCP is not a slogan.

It's a practical, concrete way to reduce that cost.

Not flashy.

Not magical.

But if adoption keeps growing, MCP could become one of the default ways AI systems connect to everything else around them.

And for anyone building real AI products, that's exactly the layer worth paying attention to — and betting on.

Because the future of AI isn't just more models.

It's more connected, more reusable, less fragmented integrations.

And MCP is the first serious attempt to make that future real.

Why Your Background Jobs Fail in Production

Damir Karimov — Mon, 29 Jun 2026 10:41:23 +0000

Most developers first meet background jobs through a deceptively simple model: put work in a queue, let a worker pick it up, and assume the job will be processed once. In production, that mental model breaks quickly because queue systems are designed for reliability under failure, not for “perfect” one-shot execution.

Need to send an email? Queue it. Generate a PDF? Queue it. Process images? Queue it.

That feels elegant at first. The request finishes fast, the slow work happens elsewhere, and everything looks stable.

Then production happens.

A worker crashes after sending the email.

A network hiccup prevents the ACK from reaching the broker.

A third-party API times out halfway through processing.

The same job gets retried and runs again.

Now you have duplicate emails, duplicate charges, or jobs that appear to vanish and then come back later. That is not an edge case. That is normal distributed-systems behavior.

The happy path is a lie

The classic flow looks clean:

User registration

→ Create user

→ Push job to queue

→ Worker processes job

→ Send email

That flow is useful for explaining the idea, but it hides the real failure modes. In practice, each step can fail independently: enqueue can succeed while downstream storage is unavailable, a worker can complete side effects and then crash before acknowledging the message, or a retry can happen because the system cannot tell whether the previous attempt fully finished.

A queue does not guarantee execution exactly once.

It coordinates work, and it does so in a failure-prone environment.

That distinction matters because it changes how you design the job handler, the data model, and the retry strategy.

At-least-once delivery

Most production queue systems follow at-least-once delivery. Amazon SQS explicitly documents that a message can be received again, and it recommends designing applications to be idempotent so repeated processing does not cause harm.

That means a job may run more than once.

For example:

Worker receives job

→ Sends email

→ Crashes before ACK

→ Queue retries job

→ Email is sent again

From the queue’s point of view, the job was never confirmed as complete. Retrying is the correct behavior. The problem is that the side effect already happened.

This is why “the queue handled it” is not the same as “the system handled it.”

Idempotency is required

Once duplicate delivery is possible, idempotency becomes mandatory. The handler should produce the same final result even if it runs multiple times with the same input.

Bad example:

account.balance += 100;

If this runs twice, the user gets credited twice.

Better:

async function processPaymentJob(job: {
  transactionId: string;
  userId: string;
  amount: number;
}) {
  const alreadyProcessed = await db.paymentEvents.findUnique({
    where: { transactionId: job.transactionId },
  });

  if (alreadyProcessed) return;

  await db.$transaction(async (tx) => {
    await tx.paymentEvents.create({
      data: {
        transactionId: job.transactionId,
        userId: job.userId,
        amount: job.amount,
      },
    });

    await tx.accounts.update({
      where: { userId: job.userId },
      data: { balance: { increment: job.amount } },
    });
  });
}

The important part is not just the findUnique check. You also need a database constraint on transactionId, plus a transactional write boundary, so two workers cannot race and apply the same effect twice. That combination is much closer to production reality than a simple in-memory guard.

A good rule: store a stable business key for every side effect, then make that key unique.

Exactly-once is a trap

Engineers often ask whether exactly-once processing is possible. In practice, the answer is usually “not in the way people imagine.”

A job may charge a customer successfully, then fail while persisting the result. Or the database write may succeed while the worker crashes before confirming the message. Either way, the system ends up with inconsistent state unless it is explicitly designed for idempotency and reconciliation.

What people call “exactly-once” in production is usually a mix of:

deduplication,
idempotent handlers,
atomic state transitions,
checkpointing,
and careful recovery logic.

The useful mental model is not “never run twice.”

It is “running twice must not break anything.”

Retries can make outages worse

Retries are necessary, but uncontrolled retries can amplify a problem.

Imagine 50,000 jobs depending on a third-party API. If that API slows down or goes offline and every worker retries immediately, you create a retry storm. Instead of letting the system recover, you add more pressure to the failing dependency.

A better pattern is exponential backoff with jitter:

Retry #1 after a short delay
Retry #2 after a longer delay
Retry #3 after an even longer delay
Add random jitter so retries do not synchronize

function getRetryDelay(attempt: number) {
  const base = Math.min(60_000 * 2 ** attempt, 3_600_000);
  const jitter = base * (0.5 + Math.random() * 0.5);
  return Math.floor(jitter);
}

Without jitter, thousands of jobs can wake up at the same time and hit the same dependency together. With jitter, the pressure spreads out, which is exactly what you want during recovery.

Dead letter queues matter early

Some jobs are never going to succeed.

Examples include:

invalid payloads,
missing records,
deleted resources,
permanent validation errors.

If a job fails for a non-transient reason, retrying it forever wastes capacity and makes the queue harder to reason about. That is what a dead letter queue is for: a place to send messages that have exceeded their retry budget or failed in a non-recoverable way.

A healthy production setup usually treats DLQ as a first-class path, not an afterthought. It gives you a clear place to inspect poison messages, track recurring failures, and keep the main queue clean.

If you do one thing after adding retries, make sure you also define where failed jobs go next.

Long jobs should be split up

A common mistake is to make one job do everything:

generate a report,
download assets,
resize images,
upload files,
update the database,
send a notification.

That looks convenient until step five fails and you have to decide whether to rerun the whole thing. The larger the job, the harder it is to retry safely and the more expensive each failure becomes.

A better shape is a chain of smaller units:

generate report,
process assets,
upload results,
notify user.

Smaller jobs are easier to retry, easier to observe, and easier to scale independently. They also reduce blast radius when something breaks.

Monitoring matters more than processing

Many teams spend a lot of time building queue workers and very little time watching them. That is a mistake.

At minimum, track:

queue length,
queue lag,
processing latency,
job duration,
failure rate,
retry volume,
DLQ growth.

These signals tell you whether workers are keeping up, whether a dependency is slowing the pipeline, and whether failures are starting to accumulate. Azure’s background-job guidance also emphasizes reliability and scaling as first-class concerns, not optional extras.

If you cannot see your queue health, you are debugging blind.

Backpressure is the silent killer

Backpressure is what happens when work arrives faster than you can process it.

If your system receives 100 jobs per second but can only complete 50, the backlog grows continuously. Eventually that leads to rising latency, memory pressure, and a degraded user experience.

Common ways to deal with it include:

rate limiting,
worker autoscaling,
priority queues,
load shedding,
rejecting or delaying non-critical work.

Ignoring backpressure does not remove it. It just makes the failure slower and more expensive.

What production systems actually look like

What tutorials often show:

Queue → Worker → Done

What production systems usually need:

App

→ Queue

→ Workers

→ Retry policy

→ Dead letter queue

→ Metrics

→ Alerting

→ Recovery process

The processing code is rarely the hardest part. The hard part is designing for failure, visibility, and recovery.

Production checklist

Before shipping a queue-based system, verify that:

The job is idempotent.
The same job can run multiple times safely.
Retry delays use exponential backoff.
Jitter is enabled.
A dead letter queue exists.
Metrics and alerts are in place.
Failure scenarios were tested intentionally.
Workers can scale horizontally.
Backpressure is handled explicitly.
Jobs are small enough to retry without fear.

If several of these are missing, the system is probably more fragile than it looks.

Final thought

Background jobs fail in production not because queues are bad, but because the happy-path mental model is incomplete.

Real systems deal with crashes, retries, duplicate delivery, slow dependencies, and poisoned messages. The goal is not to pretend those failures do not exist. The goal is to make them survivable.

Once you design for failure instead of assuming success, background jobs become much more trustworthy.

Designing a Real-Time Chat System at Scale

Damir Karimov — Tue, 23 Jun 2026 09:02:11 +0000

Most developers underestimate how hard messaging systems really are.

A basic chat demo with WebSockets is easy to build. A production-grade messaging platform like WhatsApp, Telegram, Discord, or Slack is a completely different engineering problem. The hard part is not rendering messages in the UI. The hard parts are keeping millions of persistent connections alive, delivering messages reliably, preserving ordering, handling offline sync, scaling group chats, surviving partial failures, and keeping latency low around the world.

In this post, we'll design a scalable real-time chat architecture and walk through the trade-offs behind modern messaging systems.

Why chat gets hard

At first glance, chat seems simple:

send a message
store it
show it to the other user

That works for a demo. It falls apart fast in production.

Once you add real users, the system has to handle:

persistent connections
reconnects
message ordering
offline delivery
read receipts
typing indicators
multi-device sync
large group chats
global latency
partial failures

This is where chat stops being a UI feature and becomes a distributed systems problem.

Requirements

Before designing the system, we need clear requirements.

Functional requirements

Our chat system should support:

One-to-one chats
Group chats
Real-time message delivery
Read receipts
Typing indicators
Push notifications
Media attachments
Message history
Multi-device synchronization
Online and offline presence

Non-functional requirements

The system must also provide:

Low latency
High availability
Horizontal scalability
Fault tolerance
Reliable delivery
Event ordering
Efficient storage
Global distribution

At small scale, these are manageable. At millions of users, every one of them becomes a distributed systems concern.

High-level architecture

A modern chat platform is usually event-driven and distributed.

Clients
   │
   ▼
Load Balancer
   │
   ▼
WebSocket Gateway Cluster
   │
   ├── Authentication Service
   ├── Presence Service
   ├── Chat Service
   ├── Notification Service
   └── Media Service
          │
          ▼
    Kafka / Redis Streams
          │
          ▼
     Storage Layer

Each component has a separate responsibility. That separation is what makes the system scalable and easier to evolve.

Why polling fails

Many beginner chat apps start with polling:

GET /messages every 2 seconds

This works for a demo. It fails badly at scale.

Polling creates:

massive request overhead
unnecessary database reads
increased latency
battery drain on mobile devices
poor real-time responsiveness

Now imagine 2 million connected users polling every 2 seconds. That becomes 1 million requests per second even when nobody is sending anything. Real chat systems avoid this by using persistent connections.

Why WebSockets became the default

WebSockets keep a bidirectional connection open between the client and the server.

That gives you:

near real-time communication
lower overhead
reduced latency
efficient server push
better mobile performance

The client connects once:

Client ───── persistent connection ───── Server

After that, both sides can exchange events instantly. That is the foundation of modern messaging systems.

The hidden WebSocket cost

WebSockets solve one problem and introduce another.

A persistent connection is not free. Every connected user consumes:

a TCP socket
memory buffers
heartbeat state
authentication context

At 5 million concurrent users, this is no longer a normal API problem. It becomes a connection management problem.

That is why serious chat systems build specialized gateway infrastructure.

WebSocket gateway layer

The gateway layer is responsible for:

maintaining persistent connections
authenticating users
routing events
managing heartbeats
detecting disconnects

Gateways should stay as stateless as possible. Stateless gateways are much easier to scale, replace, and recover after failures.

Authentication flow

A common flow looks like this:

User logs in via HTTP.
Backend issues a JWT token.
Client opens a WebSocket connection.
Token is validated during the handshake.
Connection is associated with a user session.

Example:

wss://chat.example.com?token=JWT

After authentication, the gateway knows which user owns the connection.

Sending a message

Now let's look at the real delivery pipeline.

Suppose User A sends:

Hello

A simplified flow is:

Client sends a message event.
Gateway validates authentication.
Chat service validates permissions.
Message is persisted in durable storage.
Event is published into Kafka.
Recipient gateway receives the event.
Message is pushed to the recipient.
Delivery ACK is generated.
Read receipt is generated later.

This looks simple on paper. In reality, every step can fail.

Why persistence comes first

A lot of beginners try this:

deliver → save later

That is dangerous.

If the server crashes before persistence, the recipient saw the message, but the database lost it. Now the system is inconsistent.

Production systems usually do:

persist → publish → deliver

Durability comes first.

Message IDs and ordering

Distributed systems do not guarantee ordering automatically.

Example:

Message A
Message B

Recipient receives:

Message B
Message A

Why does this happen? Because messages may travel through different gateway servers, queues, and network paths.

Common ordering strategies

Timestamp ordering

Simple, but unreliable. Clock drift breaks consistency.

Incremental sequence IDs

More reliable.

Example:

conversation_id: 42

messages:
1
2
3
4

This guarantees ordering inside a conversation. Most real systems only guarantee local ordering per chat. Global ordering across the entire platform is usually impossible at scale.

Database design

Messaging systems are write-heavy. A popular group chat can generate thousands of writes per second.

Typical tables:

users
conversations
conversation_members
messages
message_status
attachments

The real challenge is partitioning.

Why a single database eventually fails

A single PostgreSQL instance works at the beginning. Over time, problems show up:

write bottlenecks
storage growth
replication lag
index size explosion

At scale, systems introduce sharding.

Sharding strategy

A common strategy is:

shard by conversation_id

Benefits:

messages for the same chat stay colocated
ordering becomes easier
queries stay efficient

Bad shard keys create hotspots. For example, sharding by user_id can spread large group chats across multiple shards and make fan-out expensive.

Kafka and event streaming

Modern messaging systems are heavily event-driven.

Kafka is useful because it provides:

durable event logs
replayability
partitioned scalability
consumer groups

Instead of services calling each other directly:

Chat Service → Kafka → Consumers

Consumers may include:

delivery service
notification service
analytics
moderation
push notifications

This decouples the system and makes failures easier to isolate.

Presence system

Presence is deceptively expensive.

Tracking online, offline, typing, and last_seen for millions of users creates a huge amount of event traffic. That is why most systems isolate presence into a dedicated service.

Presence implementation

A common architecture is:

Gateway → Redis → Presence Service

Gateways periodically send heartbeats.

Example:

PING every 30 seconds

If heartbeats stop, the user is marked offline.

Redis works well here because presence data is ephemeral. Not everything belongs in a relational database.

Typing indicators

Typing indicators look trivial. They are not.

Problems include:

high event frequency
noisy updates
unnecessary fan-out

Most systems heavily throttle typing events.

Example:

User typing → emit once every 3 seconds

Without throttling, typing indicators can overload the infrastructure faster than messages.

Fan-out in group chats

Suppose a group contains 500,000 users.

One message may require 500,000 deliveries. This is called fan-out.

Large fan-out is one of the hardest messaging problems.

Fan-out strategies

Fan-out on write

The server precomputes deliveries immediately.

fast reads
expensive writes

Fan-out on read

Messages are stored once.

cheaper writes
more expensive reads

Different platforms choose different trade-offs.

Offline synchronization

Users disconnect constantly:

mobile app closed
network loss
airplane mode
battery saver

The system must synchronize missed events efficiently.

Typical approach:

fetch all events after last_sequence_id

Example:

last_seen_message = 10451

Server returns:

10452+

Incremental synchronization is critical. Full synchronization is too expensive.

Push notifications

Offline users still need notifications.

Pipeline:

message event
   ↓
notification service
   ↓
APNs / FCM
   ↓
mobile device

Push systems are eventually consistent. Notifications may arrive late, duplicated, or out of order. Clients need to tolerate that.

Multi-device sync

Modern users expect sync across:

phone
desktop
browser
tablet

Each device may keep its own session. The backend tracks:

user_id
device_id
connection_id
last_sync_state

Events are usually delivered independently per device.

Reliability guarantees

Messaging systems need clear delivery semantics.

At-most-once

Fastest. Messages may disappear.

At-least-once

Reliable. Duplicates are possible.

Exactly-once

Extremely expensive and difficult in distributed systems.

Most production systems use:

at-least-once + idempotency

That is the practical choice.

Why exactly-once is mostly marketing

Exactly-once delivery across distributed infrastructure is incredibly hard.

Network failures create ambiguity:

Did the recipient receive the message?
Did the ACK get lost?
Did the retry create a duplicate?

Sometimes the sender cannot know for sure.

That is why many systems rely on retries, deduplication, and idempotent consumers instead of true exactly-once guarantees.

Common failure scenarios

Real systems fail all the time.

Examples:

gateway crashes
Kafka partition unavailable
Redis outage
slow consumers
duplicate events
partial synchronization
network splits

Reliable systems assume failure is normal.

Scaling strategies

As traffic grows, teams usually add:

horizontal scaling for gateway nodes
partitioned Kafka topics
regional infrastructure
CDN for media
caching to reduce database pressure

The biggest bottleneck is usually not CPU.

At scale, bottlenecks are often:

network throughput
memory usage
hot partitions
connection limits
disk I/O
replication lag

Chat systems are infrastructure-heavy workloads.

What makes messaging hard

The frontend UI may look simple. Underneath is a distributed system balancing:

consistency
availability
latency
reliability
scalability

Every architectural decision introduces trade-offs. You can optimize latency, durability, throughput, or cost, but rarely all at once.

That is why messaging systems remain one of the most interesting areas in system design.

Practical takeaway

If you are building chat, do not think only in terms of sockets and message lists. Think in terms of delivery guarantees, storage strategy, ordering, recovery, and fan-out.

A chat product becomes serious very quickly. The moment you need offline sync, presence, multi-device support, and reliable delivery, you are no longer building a UI feature. You are building a distributed messaging platform.

CI/CD for Modern Applications: From Manual Deployments to Reliable Delivery Pipelines

Damir Karimov — Mon, 15 Jun 2026 10:20:05 +0000

Last year my team had a production incident because someone manually deployed without running tests. The fix took 47 minutes. That's when I realized: CI/CD isn't about automation — it's about controlling risk in software delivery.

In this article, you'll learn how to design reliable pipelines for frontend (React/Next.js) and backend systems, including:

Artifact strategies and reproducible builds
Testing pyramid (unit, integration, contract, e2e)
Safe database migrations
Production observability and DORA metrics

Plus: a production checklist you can use immediately.

Modern application development is constrained less by coding speed and more by delivery reliability. Teams can ship features quickly, but production stability breaks when deployments remain manual or partially automated. CI/CD solves this by turning software delivery into a deterministic, repeatable pipeline.

What CI/CD Actually Solves

CI/CD (Continuous Integration and Continuous Delivery/Deployment) is not a toolset — it is a system design approach for software delivery.

Core problems it addresses

Problem	Impact
Integration conflicts in branches	Merge hell, delayed releases
Unpredictable release cycles	Business uncertainty, missed SLAs
Manual deployment errors	Production incidents, data corruption
Environment drift	"Works in CI, fails in prod"
Slow rollback processes	Extended MTTR, higher user impact

Goal: Not "faster deploys", but "reliable, repeatable deploys".

CI: Continuous Integration as a Validation Layer

Continuous Integration ensures every change is validated in a controlled pipeline before merging.

A minimal CI pipeline includes:

Linting (code quality enforcement)
Unit tests
Type checking (TypeScript, etc.)
Build verification

Example flow:

feature branch → PR → CI pipeline → checks pass → merge to main

Key principle: Every commit must result in a deployable artifact after CI. Without this rule, CI becomes cosmetic.

CD: Continuous Delivery vs Continuous Deployment

These terms are often mixed but differ structurally:

Aspect	Continuous Delivery	Continuous Deployment
Build state	Always deployable	Always deployable
Deployment trigger	Manual (human approval)	Automatic (after CI success)
Release decision	Controlled (gates, approvals)	Fully automated
Best for	Regulated/large systems	Fast-moving products with strong test coverage

CD controls to include:

Approval gates
Feature flags
RBAC & audit trails
Environment-specific configuration (not baked into artifacts)

Typical CI/CD Architecture

A production-grade pipeline includes:

Code → CI → Build artifact → Store → Deploy → Monitor

Components

Layer	Examples
Source control	GitHub, GitLab
CI runner	GitHub Actions, GitLab CI, Jenkins
Build system	Docker, native builds
Artifact storage	Docker registry, S3, artifact repo
Deployment system	Kubernetes, Vercel, ECS
Monitoring	Logs, metrics (Prometheus), traces (Jaeger)

Pipeline Stages in Modern Apps

A realistic pipeline is layered validation, not linear "test → build → deploy".

1. Pre-merge checks (on PR)

ESLint / Prettier
TypeScript compilation
Unit tests
Target: < 10 minutes

2. Post-merge CI

Full test suite
Integration tests
Security scanning (SAST)
Target: < 30 minutes

3. Build stage

Docker image build
Dependency locking
Artifact versioning (semver + git SHA)

4. Deployment stage

Staging rollout
Smoke tests
Production deployment (manual or automatic)

5. Post-deployment validation

Health checks
Observability signals (logs, metrics, traces)

Pipeline Internals: Artifacts, Provenance, Reproducibility

Artifact Strategy

Store immutable artifacts: Docker images, frontend build bundles, npm tarballs
Tag by semver + git SHA, store digest
Deploy by digest (not tag) to ensure consistency

Reproducible Builds

Use Docker images with pinned base versions
Lock dependency versions (package-lock.json, pnpm-lock.yaml)
Deterministic builds (no random timestamps)

Caching & Parallelization

Cache node_modules / pnpm store
Parallelize test suites
Incremental builds (Next.js, TypeScript)

Testing Strategy: The Pyramid

Test Type	When to Run	Purpose
Unit tests	Pre-merge (PR)	Fast feedback on logic
Integration	Post-merge	API, DB, service interactions
Contract tests	Post-merge	Cross-service compatibility
E2e tests	Staging/Nightly	Full user flow validation
Performance	Nightly	Latency, throughput baselines

Flaky tests:

Measure flakiness rate (target < 1–2%)
Quarantine flaky tests (separate suite)
Retry vs fix: fix critical flakies, quarantine low-priority

CI/CD in Frontend-heavy Systems (React / Next.js)

Typical Pipeline

Install dependencies (cached)
Lint + TypeScript check
Unit tests (Jest / Vitest)
Build (Next.js build)
Static analysis (bundle size, unused deps)
Deploy (Vercel / Docker / CDN)

Key Optimizations

Cache node_modules / pnpm store
Incremental builds
Split preview deployments per PR

CI/CD for Scalable Backend Systems

Backend Pipeline Requirements

Database migration strategy
Backward compatibility checks
Blue-green or canary deployments
Queue compatibility (Kafka / RabbitMQ)

Deployment Models

Model	Use Case	Risk
Rolling	Simple services	Medium
Blue-green	Critical services, fast rollback	Low
Canary	Gradual traffic shift, metrics	Low

Safe Database Migrations in CI/CD

Migration Checklist

Add column nullable
Deploy service with backward-compatible code (read new column with fallback)
Run backfill job
Make column non-nullable
Deploy service using new column without fallback
Clean up old code/columns

Orchestration:

Migration step in pipeline
Feature flag to enable new behavior
Health gate: error rate < 0.5%, latency P95 stable

Security in CI/CD

Modern pipelines must include:

Dependency scanning (npm audit, Snyk)
Secret detection
Container vulnerability scanning
Signed artifacts (supply chain security)
SBOM (Software Bill of Materials)

CI/CD without security gates becomes an attack surface.

Observability as Final Stage of CI/CD

Delivery is incomplete without feedback loop:

Logs (structured logging)
Metrics (latency, error rate)
Traces (request flow)
Alerting (SLA violations)

Key Pipeline & Production Metrics (DORA)

Metric	Target
Lead time for changes	< 1 day
Deployment frequency	Daily or on-demand
Change failure rate	< 5%
MTTR (Mean Time to Restore)	< 1 hour
Flaky-test rate	< 1–2%

CI/CD ends where production behavior becomes visible.

Common Failure Points in CI/CD Systems

Most failures are due to design flaws, not tools:

Failure Point	Risk
No clear artifact strategy	Inconsistent production builds
Missing rollback strategy	Increased risk, extended MTTR
Overloaded pipelines	Slow feedback, reduced developer productivity
Lack of environment parity	"Works in CI, fails in prod"

Rollback should be: Version switch (traffic to previous artifact digest) — NOT manual hotfix process

Production Checklist

Before promoting to production:

✅ Artifact immutable and signed

✅ DB migration safe (nullable → backfill → non-nullable)

✅ Feature flag ready for gradual enable

✅ Health gates configured (error rate, latency)

✅ Monitoring dashboards updated (deploy events visible)

✅ Rollback plan tested (traffic switch < 5 min)

Conclusion

CI/CD is not automation of deployment. It is a system for controlling risk in software delivery.

A well-designed pipeline ensures:

Predictable releases
Consistent environments
Fast feedback loops
Safe iteration speed

Teams that implement CI/CD correctly shift from "deploying code" to "managing delivery systems".

Real-Time Notification Systems Are Harder Than Most Teams Expect

Damir Karimov — Tue, 09 Jun 2026 10:26:13 +0000

If you’ve ever thought, “It’s just a WebSocket event,” this article is for you.

Notification systems look simple on the surface, but in production they fail in annoying, expensive, and user-visible ways:

duplicate notifications
missing events
race conditions
delayed delivery
mobile disconnects
retry storms
ordering bugs
state drift across regions

The tricky part is not sending a message.

The tricky part is making sure the right user gets the right notification, in the right order, with enough reliability that the system can survive crashes, retries, and mobile networks.

The problem with “just send a WebSocket event”

A basic notification flow looks like this:

Backend → WebSocket Server → Client

That works in local dev. It even works for a while in production.

Then real traffic arrives, and the system suddenly has to handle:

reconnects
offline users
multiple devices
persistence
retries
fan-out
backpressure
push fallbacks
deduplication
ordering

At that point, your “WebSocket feature” has become a distributed messaging system.

And that is a very different problem.

Delivery semantics matter first

Before you design the system, decide what guarantees you need.

Semantics	Meaning	Use case
At-most-once	Messages may be lost, but won’t be duplicated	Low-priority updates
At-least-once	Messages won’t be lost, but may be duplicated	Payments, security alerts
Effectively-once	Duplicates are removed with dedupe logic	Critical product events

Most teams make a mistake here.

They start building transport first, then discover later that they actually needed:

idempotency keys
durable cursors
sequence numbers
replay support
acknowledgements

That is why notification systems become expensive: the real problem is not delivery, it is delivery semantics.

Why fan-out breaks systems

One event is easy.

One event to 10,000 users is not.

A single action can trigger:

feed updates
badge counter updates
push notifications
email digests
analytics events
moderation triggers

That creates:

queue amplification
retry cascades
hot partitions
uneven load
latency spikes

So the system stops being “send a notification” and becomes “shape traffic safely under failure.”

Why duplicates happen

Duplicates usually do not come from one single bug. They appear from the interaction of retries, crashes, and missing idempotency.

A common chain looks like this:

1. Message is written to Kafka
2. Consumer processes it
3. Consumer crashes before committing offset
4. Partition is reassigned
5. Another consumer reads the same message
6. User gets the notification twice

That is not random.

That is at-least-once delivery with missing deduplication.

The fix

Use:

idempotency keys
dedupe storage
sequence numbers
consumer-side protection before side effects

Ordering is harder than throughput

Most users don’t complain about 200ms of delay.

They absolutely notice this:

“Payment refunded” arrives before “Payment received”

That destroys trust immediately.

Global ordering is usually too expensive. In practice, teams often choose:

per-user ordering
per-conversation ordering
approximate ordering
causal consistency where needed

For most products, per-user ordering is the best balance.

Example: sequence numbers per user

class NotificationLog:
    def __init__(self, user_id):
        self.user_id = user_id
        self.sequence = 0

    def append(self, notification):
        self.sequence += 1
        record = {
            "user_id": self.user_id,
            "sequence": self.sequence,
            "notification": notification,
            "idempotency_key": f"{self.user_id}:{self.sequence}"
        }
        kafka.produce(topic="notifications", key=self.user_id, value=record)
        return record

This is a simple way to keep ordering stable inside a user shard.

Mobile makes everything worse

Desktop clients are relatively stable.

Mobile clients are not.

You have to deal with:

app backgrounding
battery optimization
network switching
silent disconnects
delayed push delivery
OS throttling

That’s why real systems often combine:

WebSockets for active sessions
APNs for iOS
FCM for Android
polling or pull fallback
local persistence
sync checkpoints

Important detail

APNs and FCM are not guaranteed single-delivery transport.

They can:

delay notifications
drop messages under pressure
coalesce updates
expire tokens
throttle traffic

So if the notification matters, the server still needs durable state.

A real incident example

At 3AM, an on-call engineer gets paged because one user received dozens of duplicate payment emails.

What happened?

the consumer crashed mid-batch
the offset was not committed
Kafka redelivered the same event
the email sender had no dedupe check
the user got spammed

That kind of issue is painful because it is not one bug.

It is a chain of small design decisions that only becomes visible under failure.

The practical fix

def send_notification(notification):
    idempotency_key = f"{notification.user_id}:{notification.sequence}"

    if redis.exists(f"dedupe:{idempotency_key}"):
        consumer.commit()
        return

    email_service.send(notification)
    redis.setex(f"dedupe:{idempotency_key}", 24 * 3600, "1")
    consumer.commit()

The important part is not the code style.

It is the fact that the system now assumes duplicates can happen and is built to survive them.

Observability is not optional

If you cannot observe the pipeline, you cannot debug it.

Useful signals include:

queue lag
retry count
delivery success rate
connection churn
consumer health
fan-out latency
push provider errors

The real question is not:

Did we send the event?

It is:

Can we prove the user received it?

Those are very different questions.

Metrics worth tracking

Metric	Good target
Delivery success rate	>99.5%
p99 delivery latency	<500ms
Consumer lag	low and stable
Retry rate	close to zero
Connection churn	predictable

What to do in production

A good notification system needs a runbook, not just code.

If retries spike:

Throttle producers.
Pause non-critical workers.
Increase retry backoff.
Check consumer lag.
Check push provider errors.
Rehydrate missed clients from durable state.
Replay safely with idempotency keys.

That is what makes the system operationally survivable.

The scaling path

A lot of teams go through the same evolution.

Startup

API → WebSocket Server → Client

Mid-scale

API → Kafka → Notification Workers → WebSocket Gateway Cluster → Client

High-scale

API → Multi-region event bus → regional workers → regional gateways → clients

At higher scale, the hardest problems are usually:

state distribution
per-user ordering
region routing
dedupe
offline recovery

Not CPU.

State.

What strong teams optimize for

Early teams optimize for:

speed of delivery

Strong teams optimize for:

correctness
recoverability
observability
graceful degradation
idempotency

That difference matters a lot in production.

A notification that is slightly late is usually acceptable.

A notification that is duplicated, lost, or out of order is not.

Testing the system

You should test the failure modes, not just the happy path.

Try:

dropping WebSocket connections mid-message
killing consumers during processing
simulating mobile sleep/wake cycles
forcing Kafka rebalances
replaying duplicate events
load testing fan-out spikes

If the system only works when nothing fails, it is not ready.

Final thought

Real-time notification systems look simple until scale, retries, mobile behavior, ordering, and distributed state show up.

Then they become one of the hardest backend problems in the product.

The goal is not just to send events.

The goal is to make sure the right user gets the right notification, with the right semantics, even when the system is under stress.

AI Wrappers Are Dying: Why Most AI Products Fail

Damir Karimov — Wed, 27 May 2026 12:01:10 +0000

In 2026, building an app on top of OpenAI or Anthropic is easier than ever. But wrappers are dying.

A polished UI and a few RAG pipelines can get you to launch. They will not get you lasting advantage.

OpenAI API is not a competitive moat.

Wrappers Are Dying

The first wave of AI startups was inevitable. Foundation models became powerful enough that developers could ship useful products without training models from scratch. The barrier to entry dropped dramatically.

The market filled up with wrappers.

That was not irrational. It was the fastest way to test demand and prove people would pay for AI-enabled outcomes. For many founders, a wrapper was the right starting point. It reduced time-to-market and let them focus on distribution.

But wrappers that worked for speed do not work for defensibility.

Why Wrappers Are Fragile

A wrapper around an LLM is a thin interface over someone else's intelligence.

When the underlying model improves, your product advantage shrinks. When a competitor copies your UX, your edge disappears. When the model provider ships your core feature natively, your differentiation collapses overnight.

The closer your product is to a generic interface over a foundation model, the easier it is to clone.

Three problems:

The UI is visible and easy to imitate.
The prompts and workflows are often not deeply proprietary.
The core model capability is rented, not owned.

Many AI products compete on packaging rather than infrastructure.

If your product can be described as "ChatGPT, but for X," you have product-market fit risk before you have a moat.

What Creates Real Moat

A real moat in AI is not "we use GPT." It is owning something the next startup cannot easily replicate.

That includes:

Proprietary data
Embedded workflows
Deep enterprise integration
Distribution advantages
Domain-specific expertise
Feedback loops that improve the product over time

Model access is replaceable. Workflow capture is sticky.

If your product becomes part of how a team actually works, not just a tool they try once, you build defensibility. If you own the system of record, the approval flow, the compliance layer, or the operational pipeline, you are selling infrastructure, not AI.

The more your product learns from user behavior, customer data, and domain-specific outcomes, the harder it becomes to copy.

Moat Patterns That Survive

Proprietary Data Moat

If your product collects high-signal, domain-specific data that competitors cannot access, you improve faster over time.

Examples:

labeled support cases
medical annotations
legal review outcomes
sales conversation feedback
codebase-specific assistant traces

The moat works only if the data turns into better predictions, better retrieval, or better workflow decisions.

Workflow Moat

If your product becomes the place where work starts, gets reviewed, and gets approved, switching becomes painful.

Workflow moats require:

native integrations
permissions and access control
human-in-the-loop steps
audit logs
reliable outputs that fit existing processes

Enterprise AI products win by becoming infrastructure, not assistants.

Distribution Moat

If your product is embedded in Slack, email, CRM, IDEs, or internal tooling, it becomes harder to displace. Adoption is already inside the user's daily flow.

The best model in the world loses if users never reach it.

Trust and Compliance Moat

In regulated environments, trust is product value.

If you can prove data handling, retention rules, access controls, auditability, and predictable behavior, you compete on more than output quality. For enterprise buyers, this is the difference between a demo and a contract.

Cost and Infrastructure Moat

Some AI products create advantage by reducing inference cost, latency, or operational overhead at scale.

This moat is weaker than proprietary data or workflow lock-in. It matters when usage volume is high. If you deliver similar quality at lower cost, your margin improves and pricing flexibility increases.

RAG Alone Is Not Enough

RAG is useful. It is not a moat.

Retrieval connects foundation models to private corpora, internal docs, and customer-specific context. But if every competitor can index similar documents and call the same model, the architecture is not defensible.

RAG becomes valuable when paired with:

proprietary corpora
strong ranking and retrieval quality
feedback loops
domain-specific evaluation

The moat is not the retrieval layer. It is retrieval, data quality, and embedded usage over time.

Platform Dependency Is a Liability

The biggest hidden risk in AI startups is platform dependency.

If your roadmap depends on a single provider, you inherit their pricing, latency, policy changes, rate limits, and feature roadmap. That is not a moat. That is a liability.

When OpenAI improves a capability, it helps the whole market, including your competitors. When OpenAI ships a built-in feature that overlaps with your product, your differentiation evaporates overnight.

Relying entirely on external model APIs is dangerous for long-term architecture. The more your product is a front-end to a general model, the more exposed you are to commoditization.

Ask this: if model prices change, if output quality improves, or if the model vendor ships your core feature natively, what still makes you valuable?

Enterprise Workflows Are Where Winners Live

The strongest AI products solve a workflow that already exists inside a company. They do more than "answer questions."

Enterprise buyers care about more than output quality. They care about:

Access control
Compliance
Auditability
Data retention
Integrations with existing systems
Human approval steps
Reliability at scale

Workflow-based products have stronger moats than generic assistants. They do not just generate text. They become part of operational machinery.

Once AI is embedded in billing, support, procurement, legal review, or internal knowledge systems, switching costs rise quickly.

The best products feel "boring" from the outside. They are not flashy consumer apps. They are operational systems that save time, reduce risk, or increase throughput.

Vertical AI Wins

Vertical AI is stronger than horizontal AI because it combines domain data, workflow design, and distribution.

A vertical product knows the problem deeply. It understands terminology, edge cases, compliance rules, and customer expectations in a specific domain. This makes it harder to replace with a generic chatbot.

Proprietary data becomes especially important here. The more your product learns from a narrow, high-value domain, the more its quality ties to data that others do not have.

Winners connect three things:

domain-specific data
operational workflow
recurring business value

A good vertical AI product is deeply fitted to a single job. That fit becomes harder to copy with every interaction.

Which AI Companies Survive

AI companies that survive are not the ones with the flashiest demos. They turn model capability into durable product advantage.

They:

own proprietary or hard-to-access data
sit inside critical workflows
integrate deeply into enterprise systems
build operational infrastructure, not interfaces
create switching costs through usage, trust, and process

The model may be replaceable. The product around it should not be.

This is the difference between a temporary AI app and a lasting business.

How to Measure Moat

Signals that the moat is getting stronger:

Retention stays high even when model quality changes
Customers rely on the product as part of a repeatable workflow
The cost to replicate your dataset is high
More value comes from your proprietary layer than from the base model
Integrations increase switching costs over time
Unit economics improve as usage and feedback grow

Test: if a competitor copied your UI tomorrow, would they still need the same data, trust, integrations, and operational context to match your product?

If yes, you are building a real moat.

Conclusion

The problem with most AI products is not that they use AI. They confuse access to AI with defensibility.

A great interface gets attention. It rarely creates a moat. Real technical moats come from data, workflow, infrastructure, and integration — things hard to copy and harder to unwind.

The right question is not "How can we add a model?" The right question is: What do we own that becomes more valuable over time?

The best AI companies are not the ones with the loudest demo. They are the ones whose product gets more embedded, more trusted, and more expensive to replace every quarter.

Wrappers are dying. Build a moat instead.

Why Good Abstractions Make Debugging Harder

Damir Karimov — Thu, 21 May 2026 15:03:00 +0000

Good abstractions are great when you are building software.

They are much less great when you are debugging production.

The reason is simple: abstraction hides details, and debugging often depends on the details you hoped to ignore.

In small codebases, this is barely noticeable. In real systems, especially with caches, async flows, optimistic UI, and multiple state owners, it becomes a serious problem.

The core issue

The more layers you add, the easier it is for the system to become “locally correct” and “globally wrong”.

For example:

the frontend thinks the payment succeeded,
the backend committed the transaction,
the event was published,
the cache still serves the old value,
the UI shows stale data.

Every layer is doing something reasonable.

The problem is that they are not all talking about the same version of reality.

A simple example

Imagine this flow:

User clicks Retry payment
Frontend updates UI optimistically
API returns 200 OK
Database is updated
Event is sent to downstream systems
Redis still serves old state
UI refreshes from cache and shows stale data

This is the kind of bug that wastes hours.

Not because any single line of code is hard, but because the truth is spread across several places.

Example in code

Let’s say the frontend uses optimistic updates:

const onRetryPayment = async () => {
  setPaymentStatus("PAID");

  try {
    const response = await fetch("/api/payments/retry", {
      method: "POST",
    });

    if (!response.ok) {
      throw new Error("Retry failed");
    }
  } catch (error) {
    setPaymentStatus("FAILED");
  }
};

At first glance, this looks fine.

But now imagine:

the API succeeds,
the DB is updated,
an event is emitted,
a consumer deduplicates the event incorrectly,
Redis still contains the old value,
the UI re-renders from stale cache.

The bug is no longer in this function.

The bug is in the propagation path.

Why abstractions make this worse

Abstractions hide the exact mechanics that matter during incidents.

They hide things like:

who owns the state,
when the state changes,
whether the update is synchronous or async,
whether caches are invalidated,
whether retries are safe,
whether events can arrive out of order.

That is useful in normal development.

It is terrible during debugging.

Because when something is wrong, you do not need another clean interface. You need visibility.

Typical failure patterns

These are the patterns I see most often in real systems.

1. Stale read

The data was updated, but one layer still serves an old version.

// DB updated successfully
await db.payment.update({
  where: { id: paymentId },
  data: { status: "PAID" },
});

// Cache not invalidated

Result:

DB = PAID
cache = PENDING
UI = PENDING

2. Lost update

Two writes happen close together, and one silently overwrites the other.

await updateProfile({ name: "Alex" });
await updateProfile({ name: "John" });

If the system uses last-write-wins without proper locking or versioning, the final state may not match user intent.

3. Ghost update

One layer changes, but another never receives the update.

dispatch(updateOrderStatus("PAID"));
// but query cache is never invalidated

The result is a UI that looks stuck even though the backend is correct.

4. Event reorder bug

Events arrive in a different order than they were produced.

// Event B processed before Event A
processEvent("payment_succeeded");
processEvent("payment_pending");

Now the final state may be wrong even if both handlers are valid.

The debugging trap

The trap is assuming this is a code bug.

Very often it is not.

It is a state ownership bug.

That means the real question is not:

“Which function crashed?”

The real question is:

“Which layer is the source of truth right now?”

If you cannot answer that clearly, debugging becomes guesswork.

A better way to think about it

Instead of thinking in terms of “where is the bug?”, think in terms of “where does state live?”

A useful checklist:

Where is the canonical value stored?
Which layer may cache it?
Which layer may derive it?
Which layer may overwrite it?
Which layer may delay it?
Which layer may retry it?

If the same value exists in five places, you now have five opportunities for disagreement.

Debugging strategy

When a bug crosses abstraction boundaries, I usually inspect it in this order:

Step 1: Check the source of truth

Confirm where the canonical data lives.

Step 2: Rebuild the timeline

Trace the state from user action to backend write to cache update to UI read.

Step 3: Check invalidation

If a cache exists, verify it is updated or cleared at the right moment.

Step 4: Check idempotency

If retries or events are involved, verify the operation can safely happen more than once.

Step 5: Check ordering

If events are async, verify the system does not depend on strict ordering unless it actually guarantees it.

When abstractions do help

This is not an anti-abstraction argument.

Good abstractions are still valuable when they:

reduce search space,
make ownership clear,
keep state local,
expose transitions explicitly.

For example, a small component with local state is easier to debug than three caches and two event consumers trying to keep the same value in sync.

function Counter() {
  const [count, setCount] = useState(0);

  return (
    <button onClick={() => setCount(count + 1)}>
      Count: {count}
    </button>
  );
}

This is easy to reason about because there is one owner of the state.

That is the difference.

What to do in real systems

If you want abstractions to stay helpful in production, make them observable.

That means:

add logs at boundaries,
use trace IDs,
keep ownership explicit,
invalidate caches intentionally,
design retries to be safe,
avoid hidden duplicated state.

A good abstraction should reduce complexity, not hide the mechanics that make incidents debuggable.

Final thought

The best abstractions are honest.

They do not pretend the system is simpler than it is. They make the system easier to understand without hiding where truth lives.

That is why debugging gets harder as systems grow: not because abstraction is bad, but because abstraction is often too successful at hiding the exact thing you need under pressure.

AI-generated code doesn't fail loudly. It fails correctly-looking.

Damir Karimov — Wed, 13 May 2026 12:39:58 +0000

AI-generated code rarely breaks in obvious ways. It passes review, ships
to production, and behaves correctly in controlled scenarios. The
problem is what happens after: failures appear only under timing, load,
retries, or inconsistent state transitions.

The core issue is not obvious bugs. It is code that looks structurally
correct while silently ignoring real-world failure modes.

Why AI code feels correct

AI tends to generate implementations with strong surface-level signals:

consistent TypeScript types
standard architectural patterns
clean async/await flows
readable naming conventions
familiar framework usage

This produces a strong cognitive bias during review. The code does not
look "risky", so it is assumed to be correct.

The gap appears because readability is not equivalent to correctness
under production conditions.

Where AI-generated code typically fails

1. Concurrency and race conditions

async function updateProfile(data: Profile) {
  setLoading(true);

  const response = await api.updateProfile(data);

  setUser(response.user);
  setLoading(false);
}

This assumes a single linear execution.

In real systems:

multiple requests can run in parallel
responses can resolve out of order
later responses can overwrite newer state

Result: stale state overwrite without errors or crashes.

2. Optimistic updates without consistency guarantees

setTodos((prev) => [...prev, newTodo]);
await api.createTodo(newTodo);

This assumes success.

Failure scenarios:

request fails but UI is not rolled back
retry creates duplicate entries
frontend state diverges from backend state

The system remains "visually correct" while data integrity is broken.

3. Stale closures and lifecycle assumptions

useEffect(() => {
  const interval = setInterval(() => {
    console.log(count);
  }, 1000);

  return () => clearInterval(interval);
}, []);

This pattern locks in initial state.

In production:

values become stale over time
UI desynchronization occurs
behavior depends on render timing rather than logic

No runtime error occurs, so the issue is often missed.

4. Weak caching and invalidation logic

const cacheKey = `user-${userId}`;

if (cache[cacheKey]) return cache[cacheKey];

const data = await fetchUser(userId);
cache[cacheKey] = data;
return data;

This assumes:

stable data shape
stable identity rules
single write path

In real systems:

partial updates invalidate assumptions
multiple services mutate the same entity
cache becomes silently stale rather than obviously wrong

5. Hidden assumptions about APIs

AI can introduce plausible but non-existent APIs:

await auth.refreshSession({ force: true });
storage.invalidateAllQueries();

These patterns often:

look consistent with ecosystem conventions
pass code review without deep verification
fail only at runtime

This shifts errors from compile-time to production-time.

6. Accumulated lifecycle leaks

useEffect(() => {
  const controller = new AbortController();

  fetchData(controller.signal);

  return () => controller.abort();
}, [id]);

Individually correct, but when repeated across systems:

inconsistent cleanup patterns accumulate
aborted requests still resolve in edge cases
memory usage grows gradually
behavior becomes harder to reproduce

Systemic issue: reduced verification depth

The main shift introduced by AI-generated code is not implementation
speed, but review behavior.

Before AI, writing code required reasoning during implementation. After
AI:

code already looks complete
structure appears correct by default
reviewers focus on surface validation

This creates a subtle degradation in engineering discipline:

fewer edge-case simulations
less reasoning about concurrency
weaker validation of failure states
acceptance of "looks correct" as correctness

Impact on real systems

1. Frontend state drift

UI remains stable visually while backend state diverges.

2. Authentication and session issues

race conditions during token refresh
inconsistent logout handling
background requests using invalid sessions

3. Payments and idempotency problems

duplicate transactions
retries without deduplication
partial failure inconsistencies

4. Distributed system inconsistencies

assumption of ordering guarantees
reliance on immediate consistency
incorrect retry semantics

These issues are not immediately visible. They surface as rare,
non-reproducible incidents.

The real risk

AI does not generate obviously wrong code.

It generates code that satisfies:

type safety
structural conventions
expected patterns
readable abstractions

This creates false confidence during review.

The critical failure is not bugs themselves, but reduced skepticism
toward code that appears correct.

Once that happens, correctness is no longer actively verified. It is
assumed.

Conclusion

AI increases development speed, but it also changes how correctness is
perceived.

The danger is code that looks correct enough that nobody questions it deeply.

When that happens, production issues stop being introduced by obvious mistakes and start emerging from unexamined assumptions embedded in clean-looking code.

LLM-Driven Client-Side Caching: A Hybrid Decision Architecture

Damir Karimov — Mon, 04 May 2026 15:22:22 +0000

Client-side caching is usually implemented as a storage optimization layer (TTL, SWR, invalidation rules). In practice it behaves like a decision system under uncertainty.

Static strategies fail when data volatility is non-uniform across the same application. This leads to either stale UI or excessive network traffic.

This article breaks down:

why standard caching approaches plateau
where ML improves the system
where LLMs actually fit
how to design a production-grade decision pipeline

Problem: caching is not a storage problem

Different data types behave differently:

user profiles → low volatility
feeds / notifications → high volatility
search results → context-dependent volatility
partially hydrated UI → unknown volatility

The core issue:

caching requires a policy decision per request, not a static rule

So the real problem is:

data → context → decision (cache / revalidate / bypass)

Baseline systems (what already exists)

1. SWR / TTL-based caching

Used in React Query / SWR:

stale-while-revalidate
background refetch
TTL invalidation

Works when:

update cycles are predictable
data freshness is stable

Fails when:

volatility varies inside the same dataset
freshness depends on UI state

2. Heuristic scoring systems

Example adaptive TTL:


volatilityScore = EWMA(changeFrequency)
priorityScore = userInteractionWeight * dataImportance
ttl = baseTTL / volatilityScore

Improves:

adaptive cache lifetime
frequency-aware invalidation

Limitations:

requires manual feature design
domain-specific tuning
breaks under missing signals

3. Lightweight ML models

Typical approach:

logistic regression
XGBoost / LightGBM
embedding classifiers

Pros:

fast inference
stable behavior
cheaper than LLMs

Cons:

needs labeled “optimal cache decision” data (rare)
retraining pipeline required
brittle under product changes

Why all baseline approaches plateau

All classical systems assume:

feature space is complete
behavior is stationary

In real systems:

user behavior is contextual
volatility depends on UI state
freshness is semantic, not numeric
signals are incomplete

Result:

heuristics → saturate
ML-light → overfit or drift

Key idea: caching is a decision system under uncertainty

Instead of:

“how long do we cache this?”

The correct formulation is:

“what action should we take given incomplete information?”

actions:

HIT
REVALIDATE
BYPASS
SWR

Where LLMs fit (and where they don’t)

LLMs are not a replacement layer.

They function as:

fallback policy engine for ambiguous decision space

They are useful only when:

scoring model confidence is low
signals conflict
unseen patterns appear

Architecture: layered decision system

UI Layer
   ↓
Context Builder
   ↓
Policy Engine
   ├── Rule Layer (deterministic)
   ├── ML Scoring Layer (probabilistic)
   └── LLM Fallback Layer (uncertainty)
   ↓
Cache Layer
   ↓
Network

Context model (input abstraction)

All decisions must be based on structured signals:

{
  "key": "user_feed",
  "lastUpdatedMs": 1200,
  "accessFrequency": "high",
  "volatilityScore": 0.82,
  "userAction": "scroll",
  "stalenessToleranceMs": 500
}

Important constraint:

no raw prompts
only structured features

LLM role (strictly bounded)

LLM is only a classifier:

{
  "strategy": "HIT | REVALIDATE | BYPASS | SWR",
  "ttlMs": 1200,
  "confidence": 0.78
}

Triggered only when:

ML confidence < threshold
feature signals conflict
unseen context patterns

Meta-cache: caching the decision layer

To reduce cost:

decisionCache(contextHash) → strategy

Effects:

avoids repeated LLM calls
stabilizes latency
amortizes inference cost

Cost-aware execution pipeline

IF rule matches:
    use rule engine
ELSE IF ML confidence > threshold:
    use ML model
ELSE:
    use LLM

Typical production distribution:

80–90% rules
10–20% ML
<10% LLM

Failure modes

1. Overuse of LLM

Problem:

cost spikes
unpredictable latency

Mitigation:

strict confidence gating
bounded invocation layer

2. Latency variance

Problem:

inconsistent response time in UI

Mitigation:

decision caching
async precomputation

3. Model drift

Problem:

ML decisions degrade over time

Mitigation:

feedback loop
periodic recalibration

Engineering takeaways

caching is a decision system, not storage optimization
SWR + heuristics solve majority of cases
ML-light is optimal in stable feature spaces
LLMs are only for ambiguous cases
production systems require strict routing hierarchy

Conclusion

Client-side caching becomes effective only when modeled as a layered decision system.

rules handle deterministic cases
ML handles structured uncertainty
LLM handles ambiguity

The correct design is hybrid, with strict boundaries and cost control, not LLM-centric

Discussion

Where should the boundary be defined between ML confidence and LLM fallback in production caching systems?