DEV Community: Damir Karimov

Why Your Background Jobs Fail in Production

Damir Karimov — Mon, 29 Jun 2026 10:41:23 +0000

Most developers first meet background jobs through a deceptively simple model: put work in a queue, let a worker pick it up, and assume the job will be processed once. In production, that mental model breaks quickly because queue systems are designed for reliability under failure, not for “perfect” one-shot execution.

Need to send an email? Queue it. Generate a PDF? Queue it. Process images? Queue it.

That feels elegant at first. The request finishes fast, the slow work happens elsewhere, and everything looks stable.

Then production happens.

A worker crashes after sending the email.

A network hiccup prevents the ACK from reaching the broker.

A third-party API times out halfway through processing.

The same job gets retried and runs again.

Now you have duplicate emails, duplicate charges, or jobs that appear to vanish and then come back later. That is not an edge case. That is normal distributed-systems behavior.

The happy path is a lie

The classic flow looks clean:

User registration

→ Create user

→ Push job to queue

→ Worker processes job

→ Send email

That flow is useful for explaining the idea, but it hides the real failure modes. In practice, each step can fail independently: enqueue can succeed while downstream storage is unavailable, a worker can complete side effects and then crash before acknowledging the message, or a retry can happen because the system cannot tell whether the previous attempt fully finished.

A queue does not guarantee execution exactly once.

It coordinates work, and it does so in a failure-prone environment.

That distinction matters because it changes how you design the job handler, the data model, and the retry strategy.

At-least-once delivery

Most production queue systems follow at-least-once delivery. Amazon SQS explicitly documents that a message can be received again, and it recommends designing applications to be idempotent so repeated processing does not cause harm.

That means a job may run more than once.

For example:

Worker receives job

→ Sends email

→ Crashes before ACK

→ Queue retries job

→ Email is sent again

From the queue’s point of view, the job was never confirmed as complete. Retrying is the correct behavior. The problem is that the side effect already happened.

This is why “the queue handled it” is not the same as “the system handled it.”

Idempotency is required

Once duplicate delivery is possible, idempotency becomes mandatory. The handler should produce the same final result even if it runs multiple times with the same input.

Bad example:

account.balance += 100;

If this runs twice, the user gets credited twice.

Better:

async function processPaymentJob(job: {
  transactionId: string;
  userId: string;
  amount: number;
}) {
  const alreadyProcessed = await db.paymentEvents.findUnique({
    where: { transactionId: job.transactionId },
  });

  if (alreadyProcessed) return;

  await db.$transaction(async (tx) => {
    await tx.paymentEvents.create({
      data: {
        transactionId: job.transactionId,
        userId: job.userId,
        amount: job.amount,
      },
    });

    await tx.accounts.update({
      where: { userId: job.userId },
      data: { balance: { increment: job.amount } },
    });
  });
}

The important part is not just the findUnique check. You also need a database constraint on transactionId, plus a transactional write boundary, so two workers cannot race and apply the same effect twice. That combination is much closer to production reality than a simple in-memory guard.

A good rule: store a stable business key for every side effect, then make that key unique.

Exactly-once is a trap

Engineers often ask whether exactly-once processing is possible. In practice, the answer is usually “not in the way people imagine.”

A job may charge a customer successfully, then fail while persisting the result. Or the database write may succeed while the worker crashes before confirming the message. Either way, the system ends up with inconsistent state unless it is explicitly designed for idempotency and reconciliation.

What people call “exactly-once” in production is usually a mix of:

deduplication,
idempotent handlers,
atomic state transitions,
checkpointing,
and careful recovery logic.

The useful mental model is not “never run twice.”

It is “running twice must not break anything.”

Retries can make outages worse

Retries are necessary, but uncontrolled retries can amplify a problem.

Imagine 50,000 jobs depending on a third-party API. If that API slows down or goes offline and every worker retries immediately, you create a retry storm. Instead of letting the system recover, you add more pressure to the failing dependency.

A better pattern is exponential backoff with jitter:

Retry #1 after a short delay
Retry #2 after a longer delay
Retry #3 after an even longer delay
Add random jitter so retries do not synchronize

function getRetryDelay(attempt: number) {
  const base = Math.min(60_000 * 2 ** attempt, 3_600_000);
  const jitter = base * (0.5 + Math.random() * 0.5);
  return Math.floor(jitter);
}

Without jitter, thousands of jobs can wake up at the same time and hit the same dependency together. With jitter, the pressure spreads out, which is exactly what you want during recovery.

Dead letter queues matter early

Some jobs are never going to succeed.

Examples include:

invalid payloads,
missing records,
deleted resources,
permanent validation errors.

If a job fails for a non-transient reason, retrying it forever wastes capacity and makes the queue harder to reason about. That is what a dead letter queue is for: a place to send messages that have exceeded their retry budget or failed in a non-recoverable way.

A healthy production setup usually treats DLQ as a first-class path, not an afterthought. It gives you a clear place to inspect poison messages, track recurring failures, and keep the main queue clean.

If you do one thing after adding retries, make sure you also define where failed jobs go next.

Long jobs should be split up

A common mistake is to make one job do everything:

generate a report,
download assets,
resize images,
upload files,
update the database,
send a notification.

That looks convenient until step five fails and you have to decide whether to rerun the whole thing. The larger the job, the harder it is to retry safely and the more expensive each failure becomes.

A better shape is a chain of smaller units:

generate report,
process assets,
upload results,
notify user.

Smaller jobs are easier to retry, easier to observe, and easier to scale independently. They also reduce blast radius when something breaks.

Monitoring matters more than processing

Many teams spend a lot of time building queue workers and very little time watching them. That is a mistake.

At minimum, track:

queue length,
queue lag,
processing latency,
job duration,
failure rate,
retry volume,
DLQ growth.

These signals tell you whether workers are keeping up, whether a dependency is slowing the pipeline, and whether failures are starting to accumulate. Azure’s background-job guidance also emphasizes reliability and scaling as first-class concerns, not optional extras.

If you cannot see your queue health, you are debugging blind.

Backpressure is the silent killer

Backpressure is what happens when work arrives faster than you can process it.

If your system receives 100 jobs per second but can only complete 50, the backlog grows continuously. Eventually that leads to rising latency, memory pressure, and a degraded user experience.

Common ways to deal with it include:

rate limiting,
worker autoscaling,
priority queues,
load shedding,
rejecting or delaying non-critical work.

Ignoring backpressure does not remove it. It just makes the failure slower and more expensive.

What production systems actually look like

What tutorials often show:

Queue → Worker → Done

What production systems usually need:

App

→ Queue

→ Workers

→ Retry policy

→ Dead letter queue

→ Metrics

→ Alerting

→ Recovery process

The processing code is rarely the hardest part. The hard part is designing for failure, visibility, and recovery.

Production checklist

Before shipping a queue-based system, verify that:

The job is idempotent.
The same job can run multiple times safely.
Retry delays use exponential backoff.
Jitter is enabled.
A dead letter queue exists.
Metrics and alerts are in place.
Failure scenarios were tested intentionally.
Workers can scale horizontally.
Backpressure is handled explicitly.
Jobs are small enough to retry without fear.

If several of these are missing, the system is probably more fragile than it looks.

Final thought

Background jobs fail in production not because queues are bad, but because the happy-path mental model is incomplete.

Real systems deal with crashes, retries, duplicate delivery, slow dependencies, and poisoned messages. The goal is not to pretend those failures do not exist. The goal is to make them survivable.

Once you design for failure instead of assuming success, background jobs become much more trustworthy.

Designing a Real-Time Chat System at Scale

Damir Karimov — Tue, 23 Jun 2026 09:02:11 +0000

Most developers underestimate how hard messaging systems really are.

A basic chat demo with WebSockets is easy to build. A production-grade messaging platform like WhatsApp, Telegram, Discord, or Slack is a completely different engineering problem. The hard part is not rendering messages in the UI. The hard parts are keeping millions of persistent connections alive, delivering messages reliably, preserving ordering, handling offline sync, scaling group chats, surviving partial failures, and keeping latency low around the world.

In this post, we'll design a scalable real-time chat architecture and walk through the trade-offs behind modern messaging systems.

Why chat gets hard

At first glance, chat seems simple:

send a message
store it
show it to the other user

That works for a demo. It falls apart fast in production.

Once you add real users, the system has to handle:

persistent connections
reconnects
message ordering
offline delivery
read receipts
typing indicators
multi-device sync
large group chats
global latency
partial failures

This is where chat stops being a UI feature and becomes a distributed systems problem.

Requirements

Before designing the system, we need clear requirements.

Functional requirements

Our chat system should support:

One-to-one chats
Group chats
Real-time message delivery
Read receipts
Typing indicators
Push notifications
Media attachments
Message history
Multi-device synchronization
Online and offline presence

Non-functional requirements

The system must also provide:

Low latency
High availability
Horizontal scalability
Fault tolerance
Reliable delivery
Event ordering
Efficient storage
Global distribution

At small scale, these are manageable. At millions of users, every one of them becomes a distributed systems concern.

High-level architecture

A modern chat platform is usually event-driven and distributed.

Clients
   │
   ▼
Load Balancer
   │
   ▼
WebSocket Gateway Cluster
   │
   ├── Authentication Service
   ├── Presence Service
   ├── Chat Service
   ├── Notification Service
   └── Media Service
          │
          ▼
    Kafka / Redis Streams
          │
          ▼
     Storage Layer

Each component has a separate responsibility. That separation is what makes the system scalable and easier to evolve.

Why polling fails

Many beginner chat apps start with polling:

GET /messages every 2 seconds

This works for a demo. It fails badly at scale.

Polling creates:

massive request overhead
unnecessary database reads
increased latency
battery drain on mobile devices
poor real-time responsiveness

Now imagine 2 million connected users polling every 2 seconds. That becomes 1 million requests per second even when nobody is sending anything. Real chat systems avoid this by using persistent connections.

Why WebSockets became the default

WebSockets keep a bidirectional connection open between the client and the server.

That gives you:

near real-time communication
lower overhead
reduced latency
efficient server push
better mobile performance

The client connects once:

Client ───── persistent connection ───── Server

After that, both sides can exchange events instantly. That is the foundation of modern messaging systems.

The hidden WebSocket cost

WebSockets solve one problem and introduce another.

A persistent connection is not free. Every connected user consumes:

a TCP socket
memory buffers
heartbeat state
authentication context

At 5 million concurrent users, this is no longer a normal API problem. It becomes a connection management problem.

That is why serious chat systems build specialized gateway infrastructure.

WebSocket gateway layer

The gateway layer is responsible for:

maintaining persistent connections
authenticating users
routing events
managing heartbeats
detecting disconnects

Gateways should stay as stateless as possible. Stateless gateways are much easier to scale, replace, and recover after failures.

Authentication flow

A common flow looks like this:

User logs in via HTTP.
Backend issues a JWT token.
Client opens a WebSocket connection.
Token is validated during the handshake.
Connection is associated with a user session.

Example:

wss://chat.example.com?token=JWT

After authentication, the gateway knows which user owns the connection.

Sending a message

Now let's look at the real delivery pipeline.

Suppose User A sends:

Hello

A simplified flow is:

Client sends a message event.
Gateway validates authentication.
Chat service validates permissions.
Message is persisted in durable storage.
Event is published into Kafka.
Recipient gateway receives the event.
Message is pushed to the recipient.
Delivery ACK is generated.
Read receipt is generated later.

This looks simple on paper. In reality, every step can fail.

Why persistence comes first

A lot of beginners try this:

deliver → save later

That is dangerous.

If the server crashes before persistence, the recipient saw the message, but the database lost it. Now the system is inconsistent.

Production systems usually do:

persist → publish → deliver

Durability comes first.

Message IDs and ordering

Distributed systems do not guarantee ordering automatically.

Example:

Message A
Message B

Recipient receives:

Message B
Message A

Why does this happen? Because messages may travel through different gateway servers, queues, and network paths.

Common ordering strategies

Timestamp ordering

Simple, but unreliable. Clock drift breaks consistency.

Incremental sequence IDs

More reliable.

Example:

conversation_id: 42

messages:
1
2
3
4

This guarantees ordering inside a conversation. Most real systems only guarantee local ordering per chat. Global ordering across the entire platform is usually impossible at scale.

Database design

Messaging systems are write-heavy. A popular group chat can generate thousands of writes per second.

Typical tables:

users
conversations
conversation_members
messages
message_status
attachments

The real challenge is partitioning.

Why a single database eventually fails

A single PostgreSQL instance works at the beginning. Over time, problems show up:

write bottlenecks
storage growth
replication lag
index size explosion

At scale, systems introduce sharding.

Sharding strategy

A common strategy is:

shard by conversation_id

Benefits:

messages for the same chat stay colocated
ordering becomes easier
queries stay efficient

Bad shard keys create hotspots. For example, sharding by user_id can spread large group chats across multiple shards and make fan-out expensive.

Kafka and event streaming

Modern messaging systems are heavily event-driven.

Kafka is useful because it provides:

durable event logs
replayability
partitioned scalability
consumer groups

Instead of services calling each other directly:

Chat Service → Kafka → Consumers

Consumers may include:

delivery service
notification service
analytics
moderation
push notifications

This decouples the system and makes failures easier to isolate.

Presence system

Presence is deceptively expensive.

Tracking online, offline, typing, and last_seen for millions of users creates a huge amount of event traffic. That is why most systems isolate presence into a dedicated service.

Presence implementation

A common architecture is:

Gateway → Redis → Presence Service

Gateways periodically send heartbeats.

Example:

PING every 30 seconds

If heartbeats stop, the user is marked offline.

Redis works well here because presence data is ephemeral. Not everything belongs in a relational database.

Typing indicators

Typing indicators look trivial. They are not.

Problems include:

high event frequency
noisy updates
unnecessary fan-out

Most systems heavily throttle typing events.

Example:

User typing → emit once every 3 seconds

Without throttling, typing indicators can overload the infrastructure faster than messages.

Fan-out in group chats

Suppose a group contains 500,000 users.

One message may require 500,000 deliveries. This is called fan-out.

Large fan-out is one of the hardest messaging problems.

Fan-out strategies

Fan-out on write

The server precomputes deliveries immediately.

fast reads
expensive writes

Fan-out on read

Messages are stored once.

cheaper writes
more expensive reads

Different platforms choose different trade-offs.

Offline synchronization

Users disconnect constantly:

mobile app closed
network loss
airplane mode
battery saver

The system must synchronize missed events efficiently.

Typical approach:

fetch all events after last_sequence_id

Example:

last_seen_message = 10451

Server returns:

10452+

Incremental synchronization is critical. Full synchronization is too expensive.

Push notifications

Offline users still need notifications.

Pipeline:

message event
   ↓
notification service
   ↓
APNs / FCM
   ↓
mobile device

Push systems are eventually consistent. Notifications may arrive late, duplicated, or out of order. Clients need to tolerate that.

Multi-device sync

Modern users expect sync across:

phone
desktop
browser
tablet

Each device may keep its own session. The backend tracks:

user_id
device_id
connection_id
last_sync_state

Events are usually delivered independently per device.

Reliability guarantees

Messaging systems need clear delivery semantics.

At-most-once

Fastest. Messages may disappear.

At-least-once

Reliable. Duplicates are possible.

Exactly-once

Extremely expensive and difficult in distributed systems.

Most production systems use:

at-least-once + idempotency

That is the practical choice.

Why exactly-once is mostly marketing

Exactly-once delivery across distributed infrastructure is incredibly hard.

Network failures create ambiguity:

Did the recipient receive the message?
Did the ACK get lost?
Did the retry create a duplicate?

Sometimes the sender cannot know for sure.

That is why many systems rely on retries, deduplication, and idempotent consumers instead of true exactly-once guarantees.

Common failure scenarios

Real systems fail all the time.

Examples:

gateway crashes
Kafka partition unavailable
Redis outage
slow consumers
duplicate events
partial synchronization
network splits

Reliable systems assume failure is normal.

Scaling strategies

As traffic grows, teams usually add:

horizontal scaling for gateway nodes
partitioned Kafka topics
regional infrastructure
CDN for media
caching to reduce database pressure

The biggest bottleneck is usually not CPU.

At scale, bottlenecks are often:

network throughput
memory usage
hot partitions
connection limits
disk I/O
replication lag

Chat systems are infrastructure-heavy workloads.

What makes messaging hard

The frontend UI may look simple. Underneath is a distributed system balancing:

consistency
availability
latency
reliability
scalability

Every architectural decision introduces trade-offs. You can optimize latency, durability, throughput, or cost, but rarely all at once.

That is why messaging systems remain one of the most interesting areas in system design.

Practical takeaway

If you are building chat, do not think only in terms of sockets and message lists. Think in terms of delivery guarantees, storage strategy, ordering, recovery, and fan-out.

A chat product becomes serious very quickly. The moment you need offline sync, presence, multi-device support, and reliable delivery, you are no longer building a UI feature. You are building a distributed messaging platform.

CI/CD for Modern Applications: From Manual Deployments to Reliable Delivery Pipelines

Damir Karimov — Mon, 15 Jun 2026 10:20:05 +0000

Last year my team had a production incident because someone manually deployed without running tests. The fix took 47 minutes. That's when I realized: CI/CD isn't about automation — it's about controlling risk in software delivery.

In this article, you'll learn how to design reliable pipelines for frontend (React/Next.js) and backend systems, including:

Artifact strategies and reproducible builds
Testing pyramid (unit, integration, contract, e2e)
Safe database migrations
Production observability and DORA metrics

Plus: a production checklist you can use immediately.

Modern application development is constrained less by coding speed and more by delivery reliability. Teams can ship features quickly, but production stability breaks when deployments remain manual or partially automated. CI/CD solves this by turning software delivery into a deterministic, repeatable pipeline.

What CI/CD Actually Solves

CI/CD (Continuous Integration and Continuous Delivery/Deployment) is not a toolset — it is a system design approach for software delivery.

Core problems it addresses

Problem	Impact
Integration conflicts in branches	Merge hell, delayed releases
Unpredictable release cycles	Business uncertainty, missed SLAs
Manual deployment errors	Production incidents, data corruption
Environment drift	"Works in CI, fails in prod"
Slow rollback processes	Extended MTTR, higher user impact

Goal: Not "faster deploys", but "reliable, repeatable deploys".

CI: Continuous Integration as a Validation Layer

Continuous Integration ensures every change is validated in a controlled pipeline before merging.

A minimal CI pipeline includes:

Linting (code quality enforcement)
Unit tests
Type checking (TypeScript, etc.)
Build verification

Example flow:

feature branch → PR → CI pipeline → checks pass → merge to main

Key principle: Every commit must result in a deployable artifact after CI. Without this rule, CI becomes cosmetic.

CD: Continuous Delivery vs Continuous Deployment

These terms are often mixed but differ structurally:

Aspect	Continuous Delivery	Continuous Deployment
Build state	Always deployable	Always deployable
Deployment trigger	Manual (human approval)	Automatic (after CI success)
Release decision	Controlled (gates, approvals)	Fully automated
Best for	Regulated/large systems	Fast-moving products with strong test coverage

CD controls to include:

Approval gates
Feature flags
RBAC & audit trails
Environment-specific configuration (not baked into artifacts)

Typical CI/CD Architecture

A production-grade pipeline includes:

Code → CI → Build artifact → Store → Deploy → Monitor

Components

Layer	Examples
Source control	GitHub, GitLab
CI runner	GitHub Actions, GitLab CI, Jenkins
Build system	Docker, native builds
Artifact storage	Docker registry, S3, artifact repo
Deployment system	Kubernetes, Vercel, ECS
Monitoring	Logs, metrics (Prometheus), traces (Jaeger)

Pipeline Stages in Modern Apps

A realistic pipeline is layered validation, not linear "test → build → deploy".

1. Pre-merge checks (on PR)

ESLint / Prettier
TypeScript compilation
Unit tests
Target: < 10 minutes

2. Post-merge CI

Full test suite
Integration tests
Security scanning (SAST)
Target: < 30 minutes

3. Build stage

Docker image build
Dependency locking
Artifact versioning (semver + git SHA)

4. Deployment stage

Staging rollout
Smoke tests
Production deployment (manual or automatic)

5. Post-deployment validation

Health checks
Observability signals (logs, metrics, traces)

Pipeline Internals: Artifacts, Provenance, Reproducibility

Artifact Strategy

Store immutable artifacts: Docker images, frontend build bundles, npm tarballs
Tag by semver + git SHA, store digest
Deploy by digest (not tag) to ensure consistency

Reproducible Builds

Use Docker images with pinned base versions
Lock dependency versions (package-lock.json, pnpm-lock.yaml)
Deterministic builds (no random timestamps)

Caching & Parallelization

Cache node_modules / pnpm store
Parallelize test suites
Incremental builds (Next.js, TypeScript)

Testing Strategy: The Pyramid

Test Type	When to Run	Purpose
Unit tests	Pre-merge (PR)	Fast feedback on logic
Integration	Post-merge	API, DB, service interactions
Contract tests	Post-merge	Cross-service compatibility
E2e tests	Staging/Nightly	Full user flow validation
Performance	Nightly	Latency, throughput baselines

Flaky tests:

Measure flakiness rate (target < 1–2%)
Quarantine flaky tests (separate suite)
Retry vs fix: fix critical flakies, quarantine low-priority

CI/CD in Frontend-heavy Systems (React / Next.js)

Typical Pipeline

Install dependencies (cached)
Lint + TypeScript check
Unit tests (Jest / Vitest)
Build (Next.js build)
Static analysis (bundle size, unused deps)
Deploy (Vercel / Docker / CDN)

Key Optimizations

Cache node_modules / pnpm store
Incremental builds
Split preview deployments per PR

CI/CD for Scalable Backend Systems

Backend Pipeline Requirements

Database migration strategy
Backward compatibility checks
Blue-green or canary deployments
Queue compatibility (Kafka / RabbitMQ)

Deployment Models

Model	Use Case	Risk
Rolling	Simple services	Medium
Blue-green	Critical services, fast rollback	Low
Canary	Gradual traffic shift, metrics	Low

Safe Database Migrations in CI/CD

Migration Checklist

Add column nullable
Deploy service with backward-compatible code (read new column with fallback)
Run backfill job
Make column non-nullable
Deploy service using new column without fallback
Clean up old code/columns

Orchestration:

Migration step in pipeline
Feature flag to enable new behavior
Health gate: error rate < 0.5%, latency P95 stable

Security in CI/CD

Modern pipelines must include:

Dependency scanning (npm audit, Snyk)
Secret detection
Container vulnerability scanning
Signed artifacts (supply chain security)
SBOM (Software Bill of Materials)

CI/CD without security gates becomes an attack surface.

Observability as Final Stage of CI/CD

Delivery is incomplete without feedback loop:

Logs (structured logging)
Metrics (latency, error rate)
Traces (request flow)
Alerting (SLA violations)

Key Pipeline & Production Metrics (DORA)

Metric	Target
Lead time for changes	< 1 day
Deployment frequency	Daily or on-demand
Change failure rate	< 5%
MTTR (Mean Time to Restore)	< 1 hour
Flaky-test rate	< 1–2%

CI/CD ends where production behavior becomes visible.

Common Failure Points in CI/CD Systems

Most failures are due to design flaws, not tools:

Failure Point	Risk
No clear artifact strategy	Inconsistent production builds
Missing rollback strategy	Increased risk, extended MTTR
Overloaded pipelines	Slow feedback, reduced developer productivity
Lack of environment parity	"Works in CI, fails in prod"

Rollback should be: Version switch (traffic to previous artifact digest) — NOT manual hotfix process

Production Checklist

Before promoting to production:

✅ Artifact immutable and signed

✅ DB migration safe (nullable → backfill → non-nullable)

✅ Feature flag ready for gradual enable

✅ Health gates configured (error rate, latency)

✅ Monitoring dashboards updated (deploy events visible)

✅ Rollback plan tested (traffic switch < 5 min)

Conclusion

CI/CD is not automation of deployment. It is a system for controlling risk in software delivery.

A well-designed pipeline ensures:

Predictable releases
Consistent environments
Fast feedback loops
Safe iteration speed

Teams that implement CI/CD correctly shift from "deploying code" to "managing delivery systems".

Real-Time Notification Systems Are Harder Than Most Teams Expect

Damir Karimov — Tue, 09 Jun 2026 10:26:13 +0000

If you’ve ever thought, “It’s just a WebSocket event,” this article is for you.

Notification systems look simple on the surface, but in production they fail in annoying, expensive, and user-visible ways:

duplicate notifications
missing events
race conditions
delayed delivery
mobile disconnects
retry storms
ordering bugs
state drift across regions

The tricky part is not sending a message.

The tricky part is making sure the right user gets the right notification, in the right order, with enough reliability that the system can survive crashes, retries, and mobile networks.

The problem with “just send a WebSocket event”

A basic notification flow looks like this:

Backend → WebSocket Server → Client

That works in local dev. It even works for a while in production.

Then real traffic arrives, and the system suddenly has to handle:

reconnects
offline users
multiple devices
persistence
retries
fan-out
backpressure
push fallbacks
deduplication
ordering

At that point, your “WebSocket feature” has become a distributed messaging system.

And that is a very different problem.

Delivery semantics matter first

Before you design the system, decide what guarantees you need.

Semantics	Meaning	Use case
At-most-once	Messages may be lost, but won’t be duplicated	Low-priority updates
At-least-once	Messages won’t be lost, but may be duplicated	Payments, security alerts
Effectively-once	Duplicates are removed with dedupe logic	Critical product events

Most teams make a mistake here.

They start building transport first, then discover later that they actually needed:

idempotency keys
durable cursors
sequence numbers
replay support
acknowledgements

That is why notification systems become expensive: the real problem is not delivery, it is delivery semantics.

Why fan-out breaks systems

One event is easy.

One event to 10,000 users is not.

A single action can trigger:

feed updates
badge counter updates
push notifications
email digests
analytics events
moderation triggers

That creates:

queue amplification
retry cascades
hot partitions
uneven load
latency spikes

So the system stops being “send a notification” and becomes “shape traffic safely under failure.”

Why duplicates happen

Duplicates usually do not come from one single bug. They appear from the interaction of retries, crashes, and missing idempotency.

A common chain looks like this:

1. Message is written to Kafka
2. Consumer processes it
3. Consumer crashes before committing offset
4. Partition is reassigned
5. Another consumer reads the same message
6. User gets the notification twice

That is not random.

That is at-least-once delivery with missing deduplication.

The fix

Use:

idempotency keys
dedupe storage
sequence numbers
consumer-side protection before side effects

Ordering is harder than throughput

Most users don’t complain about 200ms of delay.

They absolutely notice this:

“Payment refunded” arrives before “Payment received”

That destroys trust immediately.

Global ordering is usually too expensive. In practice, teams often choose:

per-user ordering
per-conversation ordering
approximate ordering
causal consistency where needed

For most products, per-user ordering is the best balance.

Example: sequence numbers per user

class NotificationLog:
    def __init__(self, user_id):
        self.user_id = user_id
        self.sequence = 0

    def append(self, notification):
        self.sequence += 1
        record = {
            "user_id": self.user_id,
            "sequence": self.sequence,
            "notification": notification,
            "idempotency_key": f"{self.user_id}:{self.sequence}"
        }
        kafka.produce(topic="notifications", key=self.user_id, value=record)
        return record

This is a simple way to keep ordering stable inside a user shard.

Mobile makes everything worse

Desktop clients are relatively stable.

Mobile clients are not.

You have to deal with:

app backgrounding
battery optimization
network switching
silent disconnects
delayed push delivery
OS throttling

That’s why real systems often combine:

WebSockets for active sessions
APNs for iOS
FCM for Android
polling or pull fallback
local persistence
sync checkpoints

Important detail

APNs and FCM are not guaranteed single-delivery transport.

They can:

delay notifications
drop messages under pressure
coalesce updates
expire tokens
throttle traffic

So if the notification matters, the server still needs durable state.

A real incident example

At 3AM, an on-call engineer gets paged because one user received dozens of duplicate payment emails.

What happened?

the consumer crashed mid-batch
the offset was not committed
Kafka redelivered the same event
the email sender had no dedupe check
the user got spammed

That kind of issue is painful because it is not one bug.

It is a chain of small design decisions that only becomes visible under failure.

The practical fix

def send_notification(notification):
    idempotency_key = f"{notification.user_id}:{notification.sequence}"

    if redis.exists(f"dedupe:{idempotency_key}"):
        consumer.commit()
        return

    email_service.send(notification)
    redis.setex(f"dedupe:{idempotency_key}", 24 * 3600, "1")
    consumer.commit()

The important part is not the code style.

It is the fact that the system now assumes duplicates can happen and is built to survive them.

Observability is not optional

If you cannot observe the pipeline, you cannot debug it.

Useful signals include:

queue lag
retry count
delivery success rate
connection churn
consumer health
fan-out latency
push provider errors

The real question is not:

Did we send the event?

It is:

Can we prove the user received it?

Those are very different questions.

Metrics worth tracking

Metric	Good target
Delivery success rate	>99.5%
p99 delivery latency	<500ms
Consumer lag	low and stable
Retry rate	close to zero
Connection churn	predictable

What to do in production

A good notification system needs a runbook, not just code.

If retries spike:

Throttle producers.
Pause non-critical workers.
Increase retry backoff.
Check consumer lag.
Check push provider errors.
Rehydrate missed clients from durable state.
Replay safely with idempotency keys.

That is what makes the system operationally survivable.

The scaling path

A lot of teams go through the same evolution.

Startup

API → WebSocket Server → Client

Mid-scale

API → Kafka → Notification Workers → WebSocket Gateway Cluster → Client

High-scale

API → Multi-region event bus → regional workers → regional gateways → clients

At higher scale, the hardest problems are usually:

state distribution
per-user ordering
region routing
dedupe
offline recovery

Not CPU.

State.

What strong teams optimize for

Early teams optimize for:

speed of delivery

Strong teams optimize for:

correctness
recoverability
observability
graceful degradation
idempotency

That difference matters a lot in production.

A notification that is slightly late is usually acceptable.

A notification that is duplicated, lost, or out of order is not.

Testing the system

You should test the failure modes, not just the happy path.

Try:

dropping WebSocket connections mid-message
killing consumers during processing
simulating mobile sleep/wake cycles
forcing Kafka rebalances
replaying duplicate events
load testing fan-out spikes

If the system only works when nothing fails, it is not ready.

Final thought

Real-time notification systems look simple until scale, retries, mobile behavior, ordering, and distributed state show up.

Then they become one of the hardest backend problems in the product.

The goal is not just to send events.

The goal is to make sure the right user gets the right notification, with the right semantics, even when the system is under stress.

AI Wrappers Are Dying: Why Most AI Products Fail

Damir Karimov — Wed, 27 May 2026 12:01:10 +0000

In 2026, building an app on top of OpenAI or Anthropic is easier than ever. But wrappers are dying.

A polished UI and a few RAG pipelines can get you to launch. They will not get you lasting advantage.

OpenAI API is not a competitive moat.

Wrappers Are Dying

The first wave of AI startups was inevitable. Foundation models became powerful enough that developers could ship useful products without training models from scratch. The barrier to entry dropped dramatically.

The market filled up with wrappers.

That was not irrational. It was the fastest way to test demand and prove people would pay for AI-enabled outcomes. For many founders, a wrapper was the right starting point. It reduced time-to-market and let them focus on distribution.

But wrappers that worked for speed do not work for defensibility.

Why Wrappers Are Fragile

A wrapper around an LLM is a thin interface over someone else's intelligence.

When the underlying model improves, your product advantage shrinks. When a competitor copies your UX, your edge disappears. When the model provider ships your core feature natively, your differentiation collapses overnight.

The closer your product is to a generic interface over a foundation model, the easier it is to clone.

Three problems:

The UI is visible and easy to imitate.
The prompts and workflows are often not deeply proprietary.
The core model capability is rented, not owned.

Many AI products compete on packaging rather than infrastructure.

If your product can be described as "ChatGPT, but for X," you have product-market fit risk before you have a moat.

What Creates Real Moat

A real moat in AI is not "we use GPT." It is owning something the next startup cannot easily replicate.

That includes:

Proprietary data
Embedded workflows
Deep enterprise integration
Distribution advantages
Domain-specific expertise
Feedback loops that improve the product over time

Model access is replaceable. Workflow capture is sticky.

If your product becomes part of how a team actually works, not just a tool they try once, you build defensibility. If you own the system of record, the approval flow, the compliance layer, or the operational pipeline, you are selling infrastructure, not AI.

The more your product learns from user behavior, customer data, and domain-specific outcomes, the harder it becomes to copy.

Moat Patterns That Survive

Proprietary Data Moat

If your product collects high-signal, domain-specific data that competitors cannot access, you improve faster over time.

Examples:

labeled support cases
medical annotations
legal review outcomes
sales conversation feedback
codebase-specific assistant traces

The moat works only if the data turns into better predictions, better retrieval, or better workflow decisions.

Workflow Moat

If your product becomes the place where work starts, gets reviewed, and gets approved, switching becomes painful.

Workflow moats require:

native integrations
permissions and access control
human-in-the-loop steps
audit logs
reliable outputs that fit existing processes

Enterprise AI products win by becoming infrastructure, not assistants.

Distribution Moat

If your product is embedded in Slack, email, CRM, IDEs, or internal tooling, it becomes harder to displace. Adoption is already inside the user's daily flow.

The best model in the world loses if users never reach it.

Trust and Compliance Moat

In regulated environments, trust is product value.

If you can prove data handling, retention rules, access controls, auditability, and predictable behavior, you compete on more than output quality. For enterprise buyers, this is the difference between a demo and a contract.

Cost and Infrastructure Moat

Some AI products create advantage by reducing inference cost, latency, or operational overhead at scale.

This moat is weaker than proprietary data or workflow lock-in. It matters when usage volume is high. If you deliver similar quality at lower cost, your margin improves and pricing flexibility increases.

RAG Alone Is Not Enough

RAG is useful. It is not a moat.

Retrieval connects foundation models to private corpora, internal docs, and customer-specific context. But if every competitor can index similar documents and call the same model, the architecture is not defensible.

RAG becomes valuable when paired with:

proprietary corpora
strong ranking and retrieval quality
feedback loops
domain-specific evaluation

The moat is not the retrieval layer. It is retrieval, data quality, and embedded usage over time.

Platform Dependency Is a Liability

The biggest hidden risk in AI startups is platform dependency.

If your roadmap depends on a single provider, you inherit their pricing, latency, policy changes, rate limits, and feature roadmap. That is not a moat. That is a liability.

When OpenAI improves a capability, it helps the whole market, including your competitors. When OpenAI ships a built-in feature that overlaps with your product, your differentiation evaporates overnight.

Relying entirely on external model APIs is dangerous for long-term architecture. The more your product is a front-end to a general model, the more exposed you are to commoditization.

Ask this: if model prices change, if output quality improves, or if the model vendor ships your core feature natively, what still makes you valuable?

Enterprise Workflows Are Where Winners Live

The strongest AI products solve a workflow that already exists inside a company. They do more than "answer questions."

Enterprise buyers care about more than output quality. They care about:

Access control
Compliance
Auditability
Data retention
Integrations with existing systems
Human approval steps
Reliability at scale

Workflow-based products have stronger moats than generic assistants. They do not just generate text. They become part of operational machinery.

Once AI is embedded in billing, support, procurement, legal review, or internal knowledge systems, switching costs rise quickly.

The best products feel "boring" from the outside. They are not flashy consumer apps. They are operational systems that save time, reduce risk, or increase throughput.

Vertical AI Wins

Vertical AI is stronger than horizontal AI because it combines domain data, workflow design, and distribution.

A vertical product knows the problem deeply. It understands terminology, edge cases, compliance rules, and customer expectations in a specific domain. This makes it harder to replace with a generic chatbot.

Proprietary data becomes especially important here. The more your product learns from a narrow, high-value domain, the more its quality ties to data that others do not have.

Winners connect three things:

domain-specific data
operational workflow
recurring business value

A good vertical AI product is deeply fitted to a single job. That fit becomes harder to copy with every interaction.

Which AI Companies Survive

AI companies that survive are not the ones with the flashiest demos. They turn model capability into durable product advantage.

They:

own proprietary or hard-to-access data
sit inside critical workflows
integrate deeply into enterprise systems
build operational infrastructure, not interfaces
create switching costs through usage, trust, and process

The model may be replaceable. The product around it should not be.

This is the difference between a temporary AI app and a lasting business.

How to Measure Moat

Signals that the moat is getting stronger:

Retention stays high even when model quality changes
Customers rely on the product as part of a repeatable workflow
The cost to replicate your dataset is high
More value comes from your proprietary layer than from the base model
Integrations increase switching costs over time
Unit economics improve as usage and feedback grow

Test: if a competitor copied your UI tomorrow, would they still need the same data, trust, integrations, and operational context to match your product?

If yes, you are building a real moat.

Conclusion

The problem with most AI products is not that they use AI. They confuse access to AI with defensibility.

A great interface gets attention. It rarely creates a moat. Real technical moats come from data, workflow, infrastructure, and integration — things hard to copy and harder to unwind.

The right question is not "How can we add a model?" The right question is: What do we own that becomes more valuable over time?

The best AI companies are not the ones with the loudest demo. They are the ones whose product gets more embedded, more trusted, and more expensive to replace every quarter.

Wrappers are dying. Build a moat instead.

Why Good Abstractions Make Debugging Harder

Damir Karimov — Thu, 21 May 2026 15:03:00 +0000

Good abstractions are great when you are building software.

They are much less great when you are debugging production.

The reason is simple: abstraction hides details, and debugging often depends on the details you hoped to ignore.

In small codebases, this is barely noticeable. In real systems, especially with caches, async flows, optimistic UI, and multiple state owners, it becomes a serious problem.

The core issue

The more layers you add, the easier it is for the system to become “locally correct” and “globally wrong”.

For example:

the frontend thinks the payment succeeded,
the backend committed the transaction,
the event was published,
the cache still serves the old value,
the UI shows stale data.

Every layer is doing something reasonable.

The problem is that they are not all talking about the same version of reality.

A simple example

Imagine this flow:

User clicks Retry payment
Frontend updates UI optimistically
API returns 200 OK
Database is updated
Event is sent to downstream systems
Redis still serves old state
UI refreshes from cache and shows stale data

This is the kind of bug that wastes hours.

Not because any single line of code is hard, but because the truth is spread across several places.

Example in code

Let’s say the frontend uses optimistic updates:

const onRetryPayment = async () => {
  setPaymentStatus("PAID");

  try {
    const response = await fetch("/api/payments/retry", {
      method: "POST",
    });

    if (!response.ok) {
      throw new Error("Retry failed");
    }
  } catch (error) {
    setPaymentStatus("FAILED");
  }
};

At first glance, this looks fine.

But now imagine:

the API succeeds,
the DB is updated,
an event is emitted,
a consumer deduplicates the event incorrectly,
Redis still contains the old value,
the UI re-renders from stale cache.

The bug is no longer in this function.

The bug is in the propagation path.

Why abstractions make this worse

Abstractions hide the exact mechanics that matter during incidents.

They hide things like:

who owns the state,
when the state changes,
whether the update is synchronous or async,
whether caches are invalidated,
whether retries are safe,
whether events can arrive out of order.

That is useful in normal development.

It is terrible during debugging.

Because when something is wrong, you do not need another clean interface. You need visibility.

Typical failure patterns

These are the patterns I see most often in real systems.

1. Stale read

The data was updated, but one layer still serves an old version.

// DB updated successfully
await db.payment.update({
  where: { id: paymentId },
  data: { status: "PAID" },
});

// Cache not invalidated

Result:

DB = PAID
cache = PENDING
UI = PENDING

2. Lost update

Two writes happen close together, and one silently overwrites the other.

await updateProfile({ name: "Alex" });
await updateProfile({ name: "John" });

If the system uses last-write-wins without proper locking or versioning, the final state may not match user intent.

3. Ghost update

One layer changes, but another never receives the update.

dispatch(updateOrderStatus("PAID"));
// but query cache is never invalidated

The result is a UI that looks stuck even though the backend is correct.

4. Event reorder bug

Events arrive in a different order than they were produced.

// Event B processed before Event A
processEvent("payment_succeeded");
processEvent("payment_pending");

Now the final state may be wrong even if both handlers are valid.

The debugging trap

The trap is assuming this is a code bug.

Very often it is not.

It is a state ownership bug.

That means the real question is not:

“Which function crashed?”

The real question is:

“Which layer is the source of truth right now?”

If you cannot answer that clearly, debugging becomes guesswork.

A better way to think about it

Instead of thinking in terms of “where is the bug?”, think in terms of “where does state live?”

A useful checklist:

Where is the canonical value stored?
Which layer may cache it?
Which layer may derive it?
Which layer may overwrite it?
Which layer may delay it?
Which layer may retry it?

If the same value exists in five places, you now have five opportunities for disagreement.

Debugging strategy

When a bug crosses abstraction boundaries, I usually inspect it in this order:

Step 1: Check the source of truth

Confirm where the canonical data lives.

Step 2: Rebuild the timeline

Trace the state from user action to backend write to cache update to UI read.

Step 3: Check invalidation

If a cache exists, verify it is updated or cleared at the right moment.

Step 4: Check idempotency

If retries or events are involved, verify the operation can safely happen more than once.

Step 5: Check ordering

If events are async, verify the system does not depend on strict ordering unless it actually guarantees it.

When abstractions do help

This is not an anti-abstraction argument.

Good abstractions are still valuable when they:

reduce search space,
make ownership clear,
keep state local,
expose transitions explicitly.

For example, a small component with local state is easier to debug than three caches and two event consumers trying to keep the same value in sync.

function Counter() {
  const [count, setCount] = useState(0);

  return (
    <button onClick={() => setCount(count + 1)}>
      Count: {count}
    </button>
  );
}

This is easy to reason about because there is one owner of the state.

That is the difference.

What to do in real systems

If you want abstractions to stay helpful in production, make them observable.

That means:

add logs at boundaries,
use trace IDs,
keep ownership explicit,
invalidate caches intentionally,
design retries to be safe,
avoid hidden duplicated state.

A good abstraction should reduce complexity, not hide the mechanics that make incidents debuggable.

Final thought

The best abstractions are honest.

They do not pretend the system is simpler than it is. They make the system easier to understand without hiding where truth lives.

That is why debugging gets harder as systems grow: not because abstraction is bad, but because abstraction is often too successful at hiding the exact thing you need under pressure.

AI-generated code doesn't fail loudly. It fails correctly-looking.

Damir Karimov — Wed, 13 May 2026 12:39:58 +0000

AI-generated code rarely breaks in obvious ways. It passes review, ships
to production, and behaves correctly in controlled scenarios. The
problem is what happens after: failures appear only under timing, load,
retries, or inconsistent state transitions.

The core issue is not obvious bugs. It is code that looks structurally
correct while silently ignoring real-world failure modes.

Why AI code feels correct

AI tends to generate implementations with strong surface-level signals:

consistent TypeScript types
standard architectural patterns
clean async/await flows
readable naming conventions
familiar framework usage

This produces a strong cognitive bias during review. The code does not
look "risky", so it is assumed to be correct.

The gap appears because readability is not equivalent to correctness
under production conditions.

Where AI-generated code typically fails

1. Concurrency and race conditions

async function updateProfile(data: Profile) {
  setLoading(true);

  const response = await api.updateProfile(data);

  setUser(response.user);
  setLoading(false);
}

This assumes a single linear execution.

In real systems:

multiple requests can run in parallel
responses can resolve out of order
later responses can overwrite newer state

Result: stale state overwrite without errors or crashes.

2. Optimistic updates without consistency guarantees

setTodos((prev) => [...prev, newTodo]);
await api.createTodo(newTodo);

This assumes success.

Failure scenarios:

request fails but UI is not rolled back
retry creates duplicate entries
frontend state diverges from backend state

The system remains "visually correct" while data integrity is broken.

3. Stale closures and lifecycle assumptions

useEffect(() => {
  const interval = setInterval(() => {
    console.log(count);
  }, 1000);

  return () => clearInterval(interval);
}, []);

This pattern locks in initial state.

In production:

values become stale over time
UI desynchronization occurs
behavior depends on render timing rather than logic

No runtime error occurs, so the issue is often missed.

4. Weak caching and invalidation logic

const cacheKey = `user-${userId}`;

if (cache[cacheKey]) return cache[cacheKey];

const data = await fetchUser(userId);
cache[cacheKey] = data;
return data;

This assumes:

stable data shape
stable identity rules
single write path

In real systems:

partial updates invalidate assumptions
multiple services mutate the same entity
cache becomes silently stale rather than obviously wrong

5. Hidden assumptions about APIs

AI can introduce plausible but non-existent APIs:

await auth.refreshSession({ force: true });
storage.invalidateAllQueries();

These patterns often:

look consistent with ecosystem conventions
pass code review without deep verification
fail only at runtime

This shifts errors from compile-time to production-time.

6. Accumulated lifecycle leaks

useEffect(() => {
  const controller = new AbortController();

  fetchData(controller.signal);

  return () => controller.abort();
}, [id]);

Individually correct, but when repeated across systems:

inconsistent cleanup patterns accumulate
aborted requests still resolve in edge cases
memory usage grows gradually
behavior becomes harder to reproduce

Systemic issue: reduced verification depth

The main shift introduced by AI-generated code is not implementation
speed, but review behavior.

Before AI, writing code required reasoning during implementation. After
AI:

code already looks complete
structure appears correct by default
reviewers focus on surface validation

This creates a subtle degradation in engineering discipline:

fewer edge-case simulations
less reasoning about concurrency
weaker validation of failure states
acceptance of "looks correct" as correctness

Impact on real systems

1. Frontend state drift

UI remains stable visually while backend state diverges.

2. Authentication and session issues

race conditions during token refresh
inconsistent logout handling
background requests using invalid sessions

3. Payments and idempotency problems

duplicate transactions
retries without deduplication
partial failure inconsistencies

4. Distributed system inconsistencies

assumption of ordering guarantees
reliance on immediate consistency
incorrect retry semantics

These issues are not immediately visible. They surface as rare,
non-reproducible incidents.

The real risk

AI does not generate obviously wrong code.

It generates code that satisfies:

type safety
structural conventions
expected patterns
readable abstractions

This creates false confidence during review.

The critical failure is not bugs themselves, but reduced skepticism
toward code that appears correct.

Once that happens, correctness is no longer actively verified. It is
assumed.

Conclusion

AI increases development speed, but it also changes how correctness is
perceived.

The danger is code that looks correct enough that nobody questions it deeply.

When that happens, production issues stop being introduced by obvious mistakes and start emerging from unexamined assumptions embedded in clean-looking code.

LLM-Driven Client-Side Caching: A Hybrid Decision Architecture

Damir Karimov — Mon, 04 May 2026 15:22:22 +0000

Client-side caching is usually implemented as a storage optimization layer (TTL, SWR, invalidation rules). In practice it behaves like a decision system under uncertainty.

Static strategies fail when data volatility is non-uniform across the same application. This leads to either stale UI or excessive network traffic.

This article breaks down:

why standard caching approaches plateau
where ML improves the system
where LLMs actually fit
how to design a production-grade decision pipeline

Problem: caching is not a storage problem

Different data types behave differently:

user profiles → low volatility
feeds / notifications → high volatility
search results → context-dependent volatility
partially hydrated UI → unknown volatility

The core issue:

caching requires a policy decision per request, not a static rule

So the real problem is:

data → context → decision (cache / revalidate / bypass)

Baseline systems (what already exists)

1. SWR / TTL-based caching

Used in React Query / SWR:

stale-while-revalidate
background refetch
TTL invalidation

Works when:

update cycles are predictable
data freshness is stable

Fails when:

volatility varies inside the same dataset
freshness depends on UI state

2. Heuristic scoring systems

Example adaptive TTL:


volatilityScore = EWMA(changeFrequency)
priorityScore = userInteractionWeight * dataImportance
ttl = baseTTL / volatilityScore

Improves:

adaptive cache lifetime
frequency-aware invalidation

Limitations:

requires manual feature design
domain-specific tuning
breaks under missing signals

3. Lightweight ML models

Typical approach:

logistic regression
XGBoost / LightGBM
embedding classifiers

Pros:

fast inference
stable behavior
cheaper than LLMs

Cons:

needs labeled “optimal cache decision” data (rare)
retraining pipeline required
brittle under product changes

Why all baseline approaches plateau

All classical systems assume:

feature space is complete
behavior is stationary

In real systems:

user behavior is contextual
volatility depends on UI state
freshness is semantic, not numeric
signals are incomplete

Result:

heuristics → saturate
ML-light → overfit or drift

Key idea: caching is a decision system under uncertainty

Instead of:

“how long do we cache this?”

The correct formulation is:

“what action should we take given incomplete information?”

actions:

HIT
REVALIDATE
BYPASS
SWR

Where LLMs fit (and where they don’t)

LLMs are not a replacement layer.

They function as:

fallback policy engine for ambiguous decision space

They are useful only when:

scoring model confidence is low
signals conflict
unseen patterns appear

Architecture: layered decision system

UI Layer
   ↓
Context Builder
   ↓
Policy Engine
   ├── Rule Layer (deterministic)
   ├── ML Scoring Layer (probabilistic)
   └── LLM Fallback Layer (uncertainty)
   ↓
Cache Layer
   ↓
Network

Context model (input abstraction)

All decisions must be based on structured signals:

{
  "key": "user_feed",
  "lastUpdatedMs": 1200,
  "accessFrequency": "high",
  "volatilityScore": 0.82,
  "userAction": "scroll",
  "stalenessToleranceMs": 500
}

Important constraint:

no raw prompts
only structured features

LLM role (strictly bounded)

LLM is only a classifier:

{
  "strategy": "HIT | REVALIDATE | BYPASS | SWR",
  "ttlMs": 1200,
  "confidence": 0.78
}

Triggered only when:

ML confidence < threshold
feature signals conflict
unseen context patterns

Meta-cache: caching the decision layer

To reduce cost:

decisionCache(contextHash) → strategy

Effects:

avoids repeated LLM calls
stabilizes latency
amortizes inference cost

Cost-aware execution pipeline

IF rule matches:
    use rule engine
ELSE IF ML confidence > threshold:
    use ML model
ELSE:
    use LLM

Typical production distribution:

80–90% rules
10–20% ML
<10% LLM

Failure modes

1. Overuse of LLM

Problem:

cost spikes
unpredictable latency

Mitigation:

strict confidence gating
bounded invocation layer

2. Latency variance

Problem:

inconsistent response time in UI

Mitigation:

decision caching
async precomputation

3. Model drift

Problem:

ML decisions degrade over time

Mitigation:

feedback loop
periodic recalibration

Engineering takeaways

caching is a decision system, not storage optimization
SWR + heuristics solve majority of cases
ML-light is optimal in stable feature spaces
LLMs are only for ambiguous cases
production systems require strict routing hierarchy

Conclusion

Client-side caching becomes effective only when modeled as a layered decision system.

rules handle deterministic cases
ML handles structured uncertainty
LLM handles ambiguity

The correct design is hybrid, with strict boundaries and cost control, not LLM-centric

Discussion

Where should the boundary be defined between ML confidence and LLM fallback in production caching systems?