DEV Community: Damir Karimov

Real-Time Notification Systems Are Harder Than Most Teams Expect

Damir Karimov — Tue, 09 Jun 2026 10:26:13 +0000

If you’ve ever thought, “It’s just a WebSocket event,” this article is for you.

Notification systems look simple on the surface, but in production they fail in annoying, expensive, and user-visible ways:

duplicate notifications
missing events
race conditions
delayed delivery
mobile disconnects
retry storms
ordering bugs
state drift across regions

The tricky part is not sending a message.

The tricky part is making sure the right user gets the right notification, in the right order, with enough reliability that the system can survive crashes, retries, and mobile networks.

The problem with “just send a WebSocket event”

A basic notification flow looks like this:

Backend → WebSocket Server → Client

That works in local dev. It even works for a while in production.

Then real traffic arrives, and the system suddenly has to handle:

reconnects
offline users
multiple devices
persistence
retries
fan-out
backpressure
push fallbacks
deduplication
ordering

At that point, your “WebSocket feature” has become a distributed messaging system.

And that is a very different problem.

Delivery semantics matter first

Before you design the system, decide what guarantees you need.

Semantics	Meaning	Use case
At-most-once	Messages may be lost, but won’t be duplicated	Low-priority updates
At-least-once	Messages won’t be lost, but may be duplicated	Payments, security alerts
Effectively-once	Duplicates are removed with dedupe logic	Critical product events

Most teams make a mistake here.

They start building transport first, then discover later that they actually needed:

idempotency keys
durable cursors
sequence numbers
replay support
acknowledgements

That is why notification systems become expensive: the real problem is not delivery, it is delivery semantics.

Why fan-out breaks systems

One event is easy.

One event to 10,000 users is not.

A single action can trigger:

feed updates
badge counter updates
push notifications
email digests
analytics events
moderation triggers

That creates:

queue amplification
retry cascades
hot partitions
uneven load
latency spikes

So the system stops being “send a notification” and becomes “shape traffic safely under failure.”

Why duplicates happen

Duplicates usually do not come from one single bug. They appear from the interaction of retries, crashes, and missing idempotency.

A common chain looks like this:

1. Message is written to Kafka
2. Consumer processes it
3. Consumer crashes before committing offset
4. Partition is reassigned
5. Another consumer reads the same message
6. User gets the notification twice

That is not random.

That is at-least-once delivery with missing deduplication.

The fix

Use:

idempotency keys
dedupe storage
sequence numbers
consumer-side protection before side effects

Ordering is harder than throughput

Most users don’t complain about 200ms of delay.

They absolutely notice this:

“Payment refunded” arrives before “Payment received”

That destroys trust immediately.

Global ordering is usually too expensive. In practice, teams often choose:

per-user ordering
per-conversation ordering
approximate ordering
causal consistency where needed

For most products, per-user ordering is the best balance.

Example: sequence numbers per user

class NotificationLog:
    def __init__(self, user_id):
        self.user_id = user_id
        self.sequence = 0

    def append(self, notification):
        self.sequence += 1
        record = {
            "user_id": self.user_id,
            "sequence": self.sequence,
            "notification": notification,
            "idempotency_key": f"{self.user_id}:{self.sequence}"
        }
        kafka.produce(topic="notifications", key=self.user_id, value=record)
        return record

This is a simple way to keep ordering stable inside a user shard.

Mobile makes everything worse

Desktop clients are relatively stable.

Mobile clients are not.

You have to deal with:

app backgrounding
battery optimization
network switching
silent disconnects
delayed push delivery
OS throttling

That’s why real systems often combine:

WebSockets for active sessions
APNs for iOS
FCM for Android
polling or pull fallback
local persistence
sync checkpoints

Important detail

APNs and FCM are not guaranteed single-delivery transport.

They can:

delay notifications
drop messages under pressure
coalesce updates
expire tokens
throttle traffic

So if the notification matters, the server still needs durable state.

A real incident example

At 3AM, an on-call engineer gets paged because one user received dozens of duplicate payment emails.

What happened?

the consumer crashed mid-batch
the offset was not committed
Kafka redelivered the same event
the email sender had no dedupe check
the user got spammed

That kind of issue is painful because it is not one bug.

It is a chain of small design decisions that only becomes visible under failure.

The practical fix

def send_notification(notification):
    idempotency_key = f"{notification.user_id}:{notification.sequence}"

    if redis.exists(f"dedupe:{idempotency_key}"):
        consumer.commit()
        return

    email_service.send(notification)
    redis.setex(f"dedupe:{idempotency_key}", 24 * 3600, "1")
    consumer.commit()

The important part is not the code style.

It is the fact that the system now assumes duplicates can happen and is built to survive them.

Observability is not optional

If you cannot observe the pipeline, you cannot debug it.

Useful signals include:

queue lag
retry count
delivery success rate
connection churn
consumer health
fan-out latency
push provider errors

The real question is not:

Did we send the event?

It is:

Can we prove the user received it?

Those are very different questions.

Metrics worth tracking

Metric	Good target
Delivery success rate	>99.5%
p99 delivery latency	<500ms
Consumer lag	low and stable
Retry rate	close to zero
Connection churn	predictable

What to do in production

A good notification system needs a runbook, not just code.

If retries spike:

Throttle producers.
Pause non-critical workers.
Increase retry backoff.
Check consumer lag.
Check push provider errors.
Rehydrate missed clients from durable state.
Replay safely with idempotency keys.

That is what makes the system operationally survivable.

The scaling path

A lot of teams go through the same evolution.

Startup

API → WebSocket Server → Client

Mid-scale

API → Kafka → Notification Workers → WebSocket Gateway Cluster → Client

High-scale

API → Multi-region event bus → regional workers → regional gateways → clients

At higher scale, the hardest problems are usually:

state distribution
per-user ordering
region routing
dedupe
offline recovery

Not CPU.

State.

What strong teams optimize for

Early teams optimize for:

speed of delivery

Strong teams optimize for:

correctness
recoverability
observability
graceful degradation
idempotency

That difference matters a lot in production.

A notification that is slightly late is usually acceptable.

A notification that is duplicated, lost, or out of order is not.

Testing the system

You should test the failure modes, not just the happy path.

Try:

dropping WebSocket connections mid-message
killing consumers during processing
simulating mobile sleep/wake cycles
forcing Kafka rebalances
replaying duplicate events
load testing fan-out spikes

If the system only works when nothing fails, it is not ready.

Final thought

Real-time notification systems look simple until scale, retries, mobile behavior, ordering, and distributed state show up.

Then they become one of the hardest backend problems in the product.

The goal is not just to send events.

The goal is to make sure the right user gets the right notification, with the right semantics, even when the system is under stress.

AI Wrappers Are Dying: Why Most AI Products Fail

Damir Karimov — Wed, 27 May 2026 12:01:10 +0000

In 2026, building an app on top of OpenAI or Anthropic is easier than ever. But wrappers are dying.

A polished UI and a few RAG pipelines can get you to launch. They will not get you lasting advantage.

OpenAI API is not a competitive moat.

Wrappers Are Dying

The first wave of AI startups was inevitable. Foundation models became powerful enough that developers could ship useful products without training models from scratch. The barrier to entry dropped dramatically.

The market filled up with wrappers.

That was not irrational. It was the fastest way to test demand and prove people would pay for AI-enabled outcomes. For many founders, a wrapper was the right starting point. It reduced time-to-market and let them focus on distribution.

But wrappers that worked for speed do not work for defensibility.

Why Wrappers Are Fragile

A wrapper around an LLM is a thin interface over someone else's intelligence.

When the underlying model improves, your product advantage shrinks. When a competitor copies your UX, your edge disappears. When the model provider ships your core feature natively, your differentiation collapses overnight.

The closer your product is to a generic interface over a foundation model, the easier it is to clone.

Three problems:

The UI is visible and easy to imitate.
The prompts and workflows are often not deeply proprietary.
The core model capability is rented, not owned.

Many AI products compete on packaging rather than infrastructure.

If your product can be described as "ChatGPT, but for X," you have product-market fit risk before you have a moat.

What Creates Real Moat

A real moat in AI is not "we use GPT." It is owning something the next startup cannot easily replicate.

That includes:

Proprietary data
Embedded workflows
Deep enterprise integration
Distribution advantages
Domain-specific expertise
Feedback loops that improve the product over time

Model access is replaceable. Workflow capture is sticky.

If your product becomes part of how a team actually works, not just a tool they try once, you build defensibility. If you own the system of record, the approval flow, the compliance layer, or the operational pipeline, you are selling infrastructure, not AI.

The more your product learns from user behavior, customer data, and domain-specific outcomes, the harder it becomes to copy.

Moat Patterns That Survive

Proprietary Data Moat

If your product collects high-signal, domain-specific data that competitors cannot access, you improve faster over time.

Examples:

labeled support cases
medical annotations
legal review outcomes
sales conversation feedback
codebase-specific assistant traces

The moat works only if the data turns into better predictions, better retrieval, or better workflow decisions.

Workflow Moat

If your product becomes the place where work starts, gets reviewed, and gets approved, switching becomes painful.

Workflow moats require:

native integrations
permissions and access control
human-in-the-loop steps
audit logs
reliable outputs that fit existing processes

Enterprise AI products win by becoming infrastructure, not assistants.

Distribution Moat

If your product is embedded in Slack, email, CRM, IDEs, or internal tooling, it becomes harder to displace. Adoption is already inside the user's daily flow.

The best model in the world loses if users never reach it.

Trust and Compliance Moat

In regulated environments, trust is product value.

If you can prove data handling, retention rules, access controls, auditability, and predictable behavior, you compete on more than output quality. For enterprise buyers, this is the difference between a demo and a contract.

Cost and Infrastructure Moat

Some AI products create advantage by reducing inference cost, latency, or operational overhead at scale.

This moat is weaker than proprietary data or workflow lock-in. It matters when usage volume is high. If you deliver similar quality at lower cost, your margin improves and pricing flexibility increases.

RAG Alone Is Not Enough

RAG is useful. It is not a moat.

Retrieval connects foundation models to private corpora, internal docs, and customer-specific context. But if every competitor can index similar documents and call the same model, the architecture is not defensible.

RAG becomes valuable when paired with:

proprietary corpora
strong ranking and retrieval quality
feedback loops
domain-specific evaluation

The moat is not the retrieval layer. It is retrieval, data quality, and embedded usage over time.

Platform Dependency Is a Liability

The biggest hidden risk in AI startups is platform dependency.

If your roadmap depends on a single provider, you inherit their pricing, latency, policy changes, rate limits, and feature roadmap. That is not a moat. That is a liability.

When OpenAI improves a capability, it helps the whole market, including your competitors. When OpenAI ships a built-in feature that overlaps with your product, your differentiation evaporates overnight.

Relying entirely on external model APIs is dangerous for long-term architecture. The more your product is a front-end to a general model, the more exposed you are to commoditization.

Ask this: if model prices change, if output quality improves, or if the model vendor ships your core feature natively, what still makes you valuable?

Enterprise Workflows Are Where Winners Live

The strongest AI products solve a workflow that already exists inside a company. They do more than "answer questions."

Enterprise buyers care about more than output quality. They care about:

Access control
Compliance
Auditability
Data retention
Integrations with existing systems
Human approval steps
Reliability at scale

Workflow-based products have stronger moats than generic assistants. They do not just generate text. They become part of operational machinery.

Once AI is embedded in billing, support, procurement, legal review, or internal knowledge systems, switching costs rise quickly.

The best products feel "boring" from the outside. They are not flashy consumer apps. They are operational systems that save time, reduce risk, or increase throughput.

Vertical AI Wins

Vertical AI is stronger than horizontal AI because it combines domain data, workflow design, and distribution.

A vertical product knows the problem deeply. It understands terminology, edge cases, compliance rules, and customer expectations in a specific domain. This makes it harder to replace with a generic chatbot.

Proprietary data becomes especially important here. The more your product learns from a narrow, high-value domain, the more its quality ties to data that others do not have.

Winners connect three things:

domain-specific data
operational workflow
recurring business value

A good vertical AI product is deeply fitted to a single job. That fit becomes harder to copy with every interaction.

Which AI Companies Survive

AI companies that survive are not the ones with the flashiest demos. They turn model capability into durable product advantage.

They:

own proprietary or hard-to-access data
sit inside critical workflows
integrate deeply into enterprise systems
build operational infrastructure, not interfaces
create switching costs through usage, trust, and process

The model may be replaceable. The product around it should not be.

This is the difference between a temporary AI app and a lasting business.

How to Measure Moat

Signals that the moat is getting stronger:

Retention stays high even when model quality changes
Customers rely on the product as part of a repeatable workflow
The cost to replicate your dataset is high
More value comes from your proprietary layer than from the base model
Integrations increase switching costs over time
Unit economics improve as usage and feedback grow

Test: if a competitor copied your UI tomorrow, would they still need the same data, trust, integrations, and operational context to match your product?

If yes, you are building a real moat.

Conclusion

The problem with most AI products is not that they use AI. They confuse access to AI with defensibility.

A great interface gets attention. It rarely creates a moat. Real technical moats come from data, workflow, infrastructure, and integration — things hard to copy and harder to unwind.

The right question is not "How can we add a model?" The right question is: What do we own that becomes more valuable over time?

The best AI companies are not the ones with the loudest demo. They are the ones whose product gets more embedded, more trusted, and more expensive to replace every quarter.

Wrappers are dying. Build a moat instead.

Why Good Abstractions Make Debugging Harder

Damir Karimov — Thu, 21 May 2026 15:03:00 +0000

Good abstractions are great when you are building software.

They are much less great when you are debugging production.

The reason is simple: abstraction hides details, and debugging often depends on the details you hoped to ignore.

In small codebases, this is barely noticeable. In real systems, especially with caches, async flows, optimistic UI, and multiple state owners, it becomes a serious problem.

The core issue

The more layers you add, the easier it is for the system to become “locally correct” and “globally wrong”.

For example:

the frontend thinks the payment succeeded,
the backend committed the transaction,
the event was published,
the cache still serves the old value,
the UI shows stale data.

Every layer is doing something reasonable.

The problem is that they are not all talking about the same version of reality.

A simple example

Imagine this flow:

User clicks Retry payment
Frontend updates UI optimistically
API returns 200 OK
Database is updated
Event is sent to downstream systems
Redis still serves old state
UI refreshes from cache and shows stale data

This is the kind of bug that wastes hours.

Not because any single line of code is hard, but because the truth is spread across several places.

Example in code

Let’s say the frontend uses optimistic updates:

const onRetryPayment = async () => {
  setPaymentStatus("PAID");

  try {
    const response = await fetch("/api/payments/retry", {
      method: "POST",
    });

    if (!response.ok) {
      throw new Error("Retry failed");
    }
  } catch (error) {
    setPaymentStatus("FAILED");
  }
};

At first glance, this looks fine.

But now imagine:

the API succeeds,
the DB is updated,
an event is emitted,
a consumer deduplicates the event incorrectly,
Redis still contains the old value,
the UI re-renders from stale cache.

The bug is no longer in this function.

The bug is in the propagation path.

Why abstractions make this worse

Abstractions hide the exact mechanics that matter during incidents.

They hide things like:

who owns the state,
when the state changes,
whether the update is synchronous or async,
whether caches are invalidated,
whether retries are safe,
whether events can arrive out of order.

That is useful in normal development.

It is terrible during debugging.

Because when something is wrong, you do not need another clean interface. You need visibility.

Typical failure patterns

These are the patterns I see most often in real systems.

1. Stale read

The data was updated, but one layer still serves an old version.

// DB updated successfully
await db.payment.update({
  where: { id: paymentId },
  data: { status: "PAID" },
});

// Cache not invalidated

Result:

DB = PAID
cache = PENDING
UI = PENDING

2. Lost update

Two writes happen close together, and one silently overwrites the other.

await updateProfile({ name: "Alex" });
await updateProfile({ name: "John" });

If the system uses last-write-wins without proper locking or versioning, the final state may not match user intent.

3. Ghost update

One layer changes, but another never receives the update.

dispatch(updateOrderStatus("PAID"));
// but query cache is never invalidated

The result is a UI that looks stuck even though the backend is correct.

4. Event reorder bug

Events arrive in a different order than they were produced.

// Event B processed before Event A
processEvent("payment_succeeded");
processEvent("payment_pending");

Now the final state may be wrong even if both handlers are valid.

The debugging trap

The trap is assuming this is a code bug.

Very often it is not.

It is a state ownership bug.

That means the real question is not:

“Which function crashed?”

The real question is:

“Which layer is the source of truth right now?”

If you cannot answer that clearly, debugging becomes guesswork.

A better way to think about it

Instead of thinking in terms of “where is the bug?”, think in terms of “where does state live?”

A useful checklist:

Where is the canonical value stored?
Which layer may cache it?
Which layer may derive it?
Which layer may overwrite it?
Which layer may delay it?
Which layer may retry it?

If the same value exists in five places, you now have five opportunities for disagreement.

Debugging strategy

When a bug crosses abstraction boundaries, I usually inspect it in this order:

Step 1: Check the source of truth

Confirm where the canonical data lives.

Step 2: Rebuild the timeline

Trace the state from user action to backend write to cache update to UI read.

Step 3: Check invalidation

If a cache exists, verify it is updated or cleared at the right moment.

Step 4: Check idempotency

If retries or events are involved, verify the operation can safely happen more than once.

Step 5: Check ordering

If events are async, verify the system does not depend on strict ordering unless it actually guarantees it.

When abstractions do help

This is not an anti-abstraction argument.

Good abstractions are still valuable when they:

reduce search space,
make ownership clear,
keep state local,
expose transitions explicitly.

For example, a small component with local state is easier to debug than three caches and two event consumers trying to keep the same value in sync.

function Counter() {
  const [count, setCount] = useState(0);

  return (
    <button onClick={() => setCount(count + 1)}>
      Count: {count}
    </button>
  );
}

This is easy to reason about because there is one owner of the state.

That is the difference.

What to do in real systems

If you want abstractions to stay helpful in production, make them observable.

That means:

add logs at boundaries,
use trace IDs,
keep ownership explicit,
invalidate caches intentionally,
design retries to be safe,
avoid hidden duplicated state.

A good abstraction should reduce complexity, not hide the mechanics that make incidents debuggable.

Final thought

The best abstractions are honest.

They do not pretend the system is simpler than it is. They make the system easier to understand without hiding where truth lives.

That is why debugging gets harder as systems grow: not because abstraction is bad, but because abstraction is often too successful at hiding the exact thing you need under pressure.

AI-generated code doesn't fail loudly. It fails correctly-looking.

Damir Karimov — Wed, 13 May 2026 12:39:58 +0000

AI-generated code rarely breaks in obvious ways. It passes review, ships
to production, and behaves correctly in controlled scenarios. The
problem is what happens after: failures appear only under timing, load,
retries, or inconsistent state transitions.

The core issue is not obvious bugs. It is code that looks structurally
correct while silently ignoring real-world failure modes.

Why AI code feels correct

AI tends to generate implementations with strong surface-level signals:

consistent TypeScript types
standard architectural patterns
clean async/await flows
readable naming conventions
familiar framework usage

This produces a strong cognitive bias during review. The code does not
look "risky", so it is assumed to be correct.

The gap appears because readability is not equivalent to correctness
under production conditions.

Where AI-generated code typically fails

1. Concurrency and race conditions

async function updateProfile(data: Profile) {
  setLoading(true);

  const response = await api.updateProfile(data);

  setUser(response.user);
  setLoading(false);
}

This assumes a single linear execution.

In real systems:

multiple requests can run in parallel
responses can resolve out of order
later responses can overwrite newer state

Result: stale state overwrite without errors or crashes.

2. Optimistic updates without consistency guarantees

setTodos((prev) => [...prev, newTodo]);
await api.createTodo(newTodo);

This assumes success.

Failure scenarios:

request fails but UI is not rolled back
retry creates duplicate entries
frontend state diverges from backend state

The system remains "visually correct" while data integrity is broken.

3. Stale closures and lifecycle assumptions

useEffect(() => {
  const interval = setInterval(() => {
    console.log(count);
  }, 1000);

  return () => clearInterval(interval);
}, []);

This pattern locks in initial state.

In production:

values become stale over time
UI desynchronization occurs
behavior depends on render timing rather than logic

No runtime error occurs, so the issue is often missed.

4. Weak caching and invalidation logic

const cacheKey = `user-${userId}`;

if (cache[cacheKey]) return cache[cacheKey];

const data = await fetchUser(userId);
cache[cacheKey] = data;
return data;

This assumes:

stable data shape
stable identity rules
single write path

In real systems:

partial updates invalidate assumptions
multiple services mutate the same entity
cache becomes silently stale rather than obviously wrong

5. Hidden assumptions about APIs

AI can introduce plausible but non-existent APIs:

await auth.refreshSession({ force: true });
storage.invalidateAllQueries();

These patterns often:

look consistent with ecosystem conventions
pass code review without deep verification
fail only at runtime

This shifts errors from compile-time to production-time.

6. Accumulated lifecycle leaks

useEffect(() => {
  const controller = new AbortController();

  fetchData(controller.signal);

  return () => controller.abort();
}, [id]);

Individually correct, but when repeated across systems:

inconsistent cleanup patterns accumulate
aborted requests still resolve in edge cases
memory usage grows gradually
behavior becomes harder to reproduce

Systemic issue: reduced verification depth

The main shift introduced by AI-generated code is not implementation
speed, but review behavior.

Before AI, writing code required reasoning during implementation. After
AI:

code already looks complete
structure appears correct by default
reviewers focus on surface validation

This creates a subtle degradation in engineering discipline:

fewer edge-case simulations
less reasoning about concurrency
weaker validation of failure states
acceptance of "looks correct" as correctness

Impact on real systems

1. Frontend state drift

UI remains stable visually while backend state diverges.

2. Authentication and session issues

race conditions during token refresh
inconsistent logout handling
background requests using invalid sessions

3. Payments and idempotency problems

duplicate transactions
retries without deduplication
partial failure inconsistencies

4. Distributed system inconsistencies

assumption of ordering guarantees
reliance on immediate consistency
incorrect retry semantics

These issues are not immediately visible. They surface as rare,
non-reproducible incidents.

The real risk

AI does not generate obviously wrong code.

It generates code that satisfies:

type safety
structural conventions
expected patterns
readable abstractions

This creates false confidence during review.

The critical failure is not bugs themselves, but reduced skepticism
toward code that appears correct.

Once that happens, correctness is no longer actively verified. It is
assumed.

Conclusion

AI increases development speed, but it also changes how correctness is
perceived.

The danger is code that looks correct enough that nobody questions it deeply.

When that happens, production issues stop being introduced by obvious mistakes and start emerging from unexamined assumptions embedded in clean-looking code.

LLM-Driven Client-Side Caching: A Hybrid Decision Architecture

Damir Karimov — Mon, 04 May 2026 15:22:22 +0000

Client-side caching is usually implemented as a storage optimization layer (TTL, SWR, invalidation rules). In practice it behaves like a decision system under uncertainty.

Static strategies fail when data volatility is non-uniform across the same application. This leads to either stale UI or excessive network traffic.

This article breaks down:

why standard caching approaches plateau
where ML improves the system
where LLMs actually fit
how to design a production-grade decision pipeline

Problem: caching is not a storage problem

Different data types behave differently:

user profiles → low volatility
feeds / notifications → high volatility
search results → context-dependent volatility
partially hydrated UI → unknown volatility

The core issue:

caching requires a policy decision per request, not a static rule

So the real problem is:

data → context → decision (cache / revalidate / bypass)

Baseline systems (what already exists)

1. SWR / TTL-based caching

Used in React Query / SWR:

stale-while-revalidate
background refetch
TTL invalidation

Works when:

update cycles are predictable
data freshness is stable

Fails when:

volatility varies inside the same dataset
freshness depends on UI state

2. Heuristic scoring systems

Example adaptive TTL:


volatilityScore = EWMA(changeFrequency)
priorityScore = userInteractionWeight * dataImportance
ttl = baseTTL / volatilityScore

Improves:

adaptive cache lifetime
frequency-aware invalidation

Limitations:

requires manual feature design
domain-specific tuning
breaks under missing signals

3. Lightweight ML models

Typical approach:

logistic regression
XGBoost / LightGBM
embedding classifiers

Pros:

fast inference
stable behavior
cheaper than LLMs

Cons:

needs labeled “optimal cache decision” data (rare)
retraining pipeline required
brittle under product changes

Why all baseline approaches plateau

All classical systems assume:

feature space is complete
behavior is stationary

In real systems:

user behavior is contextual
volatility depends on UI state
freshness is semantic, not numeric
signals are incomplete

Result:

heuristics → saturate
ML-light → overfit or drift

Key idea: caching is a decision system under uncertainty

Instead of:

“how long do we cache this?”

The correct formulation is:

“what action should we take given incomplete information?”

actions:

HIT
REVALIDATE
BYPASS
SWR

Where LLMs fit (and where they don’t)

LLMs are not a replacement layer.

They function as:

fallback policy engine for ambiguous decision space

They are useful only when:

scoring model confidence is low
signals conflict
unseen patterns appear

Architecture: layered decision system

UI Layer
   ↓
Context Builder
   ↓
Policy Engine
   ├── Rule Layer (deterministic)
   ├── ML Scoring Layer (probabilistic)
   └── LLM Fallback Layer (uncertainty)
   ↓
Cache Layer
   ↓
Network

Context model (input abstraction)

All decisions must be based on structured signals:

{
  "key": "user_feed",
  "lastUpdatedMs": 1200,
  "accessFrequency": "high",
  "volatilityScore": 0.82,
  "userAction": "scroll",
  "stalenessToleranceMs": 500
}

Important constraint:

no raw prompts
only structured features

LLM role (strictly bounded)

LLM is only a classifier:

{
  "strategy": "HIT | REVALIDATE | BYPASS | SWR",
  "ttlMs": 1200,
  "confidence": 0.78
}

Triggered only when:

ML confidence < threshold
feature signals conflict
unseen context patterns

Meta-cache: caching the decision layer

To reduce cost:

decisionCache(contextHash) → strategy

Effects:

avoids repeated LLM calls
stabilizes latency
amortizes inference cost

Cost-aware execution pipeline

IF rule matches:
    use rule engine
ELSE IF ML confidence > threshold:
    use ML model
ELSE:
    use LLM

Typical production distribution:

80–90% rules
10–20% ML
<10% LLM

Failure modes

1. Overuse of LLM

Problem:

cost spikes
unpredictable latency

Mitigation:

strict confidence gating
bounded invocation layer

2. Latency variance

Problem:

inconsistent response time in UI

Mitigation:

decision caching
async precomputation

3. Model drift

Problem:

ML decisions degrade over time

Mitigation:

feedback loop
periodic recalibration

Engineering takeaways

caching is a decision system, not storage optimization
SWR + heuristics solve majority of cases
ML-light is optimal in stable feature spaces
LLMs are only for ambiguous cases
production systems require strict routing hierarchy

Conclusion

Client-side caching becomes effective only when modeled as a layered decision system.

rules handle deterministic cases
ML handles structured uncertainty
LLM handles ambiguity

The correct design is hybrid, with strict boundaries and cost control, not LLM-centric

Discussion

Where should the boundary be defined between ML confidence and LLM fallback in production caching systems?