DEV Community: Tasos Nikolaou

Schrödinger's Service: Why Your Microservice Is Both Up and Down Until You Observe It

Tasos Nikolaou — Tue, 14 Jul 2026 10:28:25 +0000

Your health check returns 200.

Your users are getting errors.

Both things are true at the same time.

Welcome to Schrödinger's Service.

The Thought Experiment

In 1935, Schrödinger described a cat in a box that is simultaneously alive and dead — until you open the box and observe it.

The point wasn't the cat. The point was this:

The act of observation determines the state.

In quantum mechanics, a system doesn't have a definite state until it's measured. Before measurement, it exists in a superposition of possible states.

Your microservices do the same thing.

The Problem: You're Observing the Wrong Thing

A health check is an observation. But it's a very specific one.

It tells you: "the service responded to this particular request, at this particular moment, on this particular endpoint."

It does not tell you:

whether the database connection pool is exhausted
whether downstream dependencies are timing out
whether the cache is serving stale data
whether a specific code path that runs once every 10 minutes is broken
whether memory is slowly leaking under load

The service is in superposition. It's up and down, depending on which dimension you measure.

Your health check collapses that superposition — but it collapses it into the state you chose to observe, not the state your users experience.

What Observers Actually Collapse

In quantum mechanics, different measurements reveal different properties. You can measure position or momentum, but not both precisely at the same time (Heisenberg uncertainty principle — a topic for another article 😅).

In distributed systems, different probes reveal different failure modes:

Observation	What it collapses	What it misses
`/health` ping	Process is alive	Everything else
DB connection check	Can open a connection	Pool exhaustion, slow queries
Dependency ping	Dependency responds	Degraded responses, timeouts
Synthetic transaction	Happy path works	Edge cases, specific user data
Error rate metric	Something is failing	What, and for whom

No single observation gives you the full picture.

The state of your service is not a single value. It's a probability distribution across many dimensions. And you're sampling only a few of them.

The Observer Effect

In physics, the observer effect means that measuring a system can change it.

Shine light on an electron to locate it, and you transfer momentum. The measurement perturbs the system.

Your monitoring does this too.

A health check that opens a database connection consumes a connection from the pool. Under normal conditions, negligible. Under load, with an already-exhausted pool, your health check is now competing with real traffic for resources.

More subtle: a service that runs fine under monitoring traffic — periodic, lightweight, predictable — can fail under production traffic — bursty, heavy, unpredictable. You've been observing a different system than the one your users are using.

Superposition in Practice

Here are three real-world patterns where a service is simultaneously healthy and broken:

1. The warm path is fine, the cold path is broken

Your health check hits the same cached response every time. A rarely-executed code path — triggered only by specific user conditions — has been broken for two weeks. Your health check has no idea.

2. The replica is fine, the primary is not

Your health check reads from a replica. Writes are failing silently. Replication lag is growing. GET /health → 200. Users → errors.

3. The process is alive, the logic is not

The service is up. It's processing requests. But it started consuming a config value from the wrong environment. Every response it sends is subtly wrong. Not a crash. Not a timeout. Wrong.

Health check: green. Reality: broken.

Collapsing Into the Right State

The fix isn't better health checks. It's accepting that you need multiple, independent observations that together describe the service more completely.

Liveness — Is the process alive and not deadlocked?
Simple. Lightweight. Just a ping. This is the Schrödinger question: is it dead or alive?

Readiness — Can it serve traffic right now?
Checks immediate dependencies, connection pool availability, circuit breaker states.

Saturation — Is it approaching its limits?
Memory, connection counts, queue depth, thread pool utilization. The warning signs that appear before failure.

Correctness — Is it producing the right results?
Synthetic transactions that exercise real logic with known inputs and expected outputs. This is the hardest one, and the one most teams skip.

User-facing signals — What are real users actually experiencing?
Error rates by endpoint. Latency percentiles. Business metrics. The only observation that truly matters.

Each of these collapses the superposition differently. Together, they give you a much more complete picture of the actual state.

The Uncertainty Budget

You will never fully observe your system. There's always a dimension you haven't measured. A failure mode you haven't anticipated.

This is the engineering equivalent of the uncertainty principle: you can reduce uncertainty, but not eliminate it.

What you can do is be explicit about your observability budget:

Which states do you observe?
Which states are you blind to?
What are the consequences if those unobserved states are broken?

Most teams don't answer these questions. They add a health check, mark the service as monitored, and move on.

But an unobserved failure is still a failure.

The cat doesn't care whether you've opened the box!

Practical Checklist

Before marking a service as "observable":

Liveness probe: Is the process alive?
Readiness probe: Can it serve traffic? Does it check real dependencies?
Saturation metrics: Memory, connections, queue depth — with alerting thresholds
Synthetic transaction: Does a real user flow actually work end-to-end?
User-facing error rate: Are real requests succeeding?
Rare code paths: When did you last test the paths your health check never touches?

If you can't answer all six, your service is still in superposition.

Conclusion

Schrödinger's cat isn't a paradox about cats. It's a statement about the limits of observation.

Your microservice isn't healthy because your health check says so. It's healthy because you've constructed enough independent observations to be confident that the state your users experience matches the state you're measuring.

That's a much higher bar.

Most services never reach it.

A 200 from your health check doesn't mean the service is up.
It means the service was up, for that probe, at that moment.
Everything else is superposition.

There Is No "Now" in Distributed Systems

Tasos Nikolaou — Wed, 08 Jul 2026 07:31:49 +0000

Ask your two services what time it is.

They'll give you different answers.

Both will be wrong.

The Relativity Problem

In 1905, Einstein published special relativity. One of its core conclusions was uncomfortable:

There is no universal "now."

Two observers moving relative to each other will disagree about whether two events happened simultaneously. Not because of measurement error. Not because of imprecise clocks. Because simultaneity itself is relative.

Whether two things happened "at the same time" depends entirely on who's asking.

I kept thinking about this while debugging a distributed system.

Service A logs: "order submitted at 14:23:00.412"
Service B logs: "payment processed at 14:23:00.408"

Payment before submission. Four milliseconds of causality violation.

No hardware failure. No bug. Just clocks.

What NTP Actually Guarantees

Most distributed systems assume their clocks are synchronized. They use NTP. They trust the timestamps.

Here's what NTP actually gives you: clocks that are probably within 1–100 milliseconds of each other, under normal network conditions.

That's not "synchronized." That's "close enough for most things."

It is not close enough for:

establishing event ordering across services
audit logs that need to reconstruct causality
distributed transactions
anything where "which happened first" matters

Clock drift is real. Network delays are asymmetric. A server that was briefly partitioned and reconnected can have a clock that jumped forward or backward. NTP corrects this gradually — meaning for a window of time, your clock is knowingly wrong while it adjusts.

The assumption that "timestamp A < timestamp B" means "A happened before B" is false in distributed systems.

It's not an edge case. It's the default.

Einstein's Lesson: Define Your Reference Frame

In relativity, the fix isn't to find the "true" time. There is no universal reference frame. Instead, you define your frame explicitly, and work within it.

Statements like "A happened before B" are only meaningful when you specify: according to whom?

Distributed systems need the same discipline.

The question isn't "what time did this happen?" The question is:

"What is the causal relationship between these events?"

Those are different questions. And only the second one has a reliable answer.

Logical Clocks: Causality Without Clocks

Leslie Lamport solved this in 1978 with logical clocks.

The insight: you don't need wall clock time to establish ordering. You need causality.

Lamport timestamps assign a counter to each event. The rule is simple:

Every event increments your local counter
Every message carries the sender's counter
On receipt, you take the maximum of your counter and the received one, then increment

The result: if A causally precedes B (A's message caused B to happen), then A's timestamp is guaranteed to be less than B's.

What it doesn't give you: if A's timestamp < B's timestamp, that doesn't mean A caused B. They might be concurrent — unrelated events that happened independently.

Vector clocks go further. Instead of a single counter, each service maintains a vector of counters, one per service. Now you can detect true concurrency: two events are concurrent if neither causally precedes the other.

This is how systems like DynamoDB, Riak, and CRDTs handle conflicting writes. Not by assuming clocks, but by tracking causality explicitly.

The Speed of Light Problem

In general relativity, nothing travels faster than light. Information takes time to propagate. Two observers far apart cannot instantly agree on a shared state.

In distributed systems, the equivalent is network latency. There is a speed-of-light limit on consensus.

This is what the CAP theorem is actually saying:

During a network partition, you cannot have both consistency and availability.

Choose consistency: all nodes agree on the current state, but you have to wait (or refuse requests) during partition.

Choose availability: every node responds, but they might disagree on what the current state is.

There is no third option. Physics doesn't have a third option either.

The engineers who understand this don't fight it. They design around it. They ask: for this specific piece of state, what consistency level do I actually need?

User session token → strong consistency (can't be in two places)
Shopping cart → eventual consistency (merge conflicts are recoverable)
View count → approximate is fine
Financial ledger → strong consistency with auditability

Different data has different consistency requirements. Treating everything as strongly consistent is expensive and slow. Treating everything as eventually consistent is dangerous.

When "Now" Actually Matters

Some systems need to know the real wall clock time. Financial transactions, audit logs, certificate issuance. You can't replace wall time with logical clocks in these cases.

The honest answer is that you need bounded uncertainty.

Google's TrueTime API (used in Spanner) is the best production example. Instead of returning a timestamp, it returns an interval: [earliest, latest]. The API commits that the true time lies somewhere in that interval, with the uncertainty being at most a few milliseconds.

Operations wait until uncertainty windows don't overlap before committing. The system is correct, but it explicitly accounts for the fact that "now" is not a point — it's a range.

Most of us don't have atomic clocks and GPS receivers in our datacenters. But the mental model applies:

When you see a timestamp, think: this is a range, not a point. What happens to your system if the true time is at the edges of that range?

Practical Consequences

Things that break when you assume wall clocks are synchronized:

Distributed locks with TTLs — If Service A acquires a lock that expires in 10 seconds, and its clock runs 50ms fast, the lock expires slightly earlier than expected. Under normal conditions, fine. Under load, with GC pauses and clock adjustments, your "held" lock has already expired.

Event sourcing and replay — If you reconstruct state by replaying events ordered by timestamp, clock skew means events can replay in the wrong order. The reconstructed state is wrong.

Log correlation — Joining logs from two services by timestamp to reconstruct a request's journey gives you a fuzzy picture at best. Events within the NTP uncertainty window cannot be reliably ordered.

Cache invalidation with TTLs — A cache entry set at T expires at T+300s. If the cache server and the origin server have drifted clocks, "expired" is ambiguous.

What To Do Instead

For ordering: use logical clocks (Lamport or vector) when causality matters. Don't use wall time.

For distributed state: embrace eventual consistency where you can. Design conflict resolution (last-write-wins, CRDTs, explicit merge functions) where you can't.

For wall time: treat timestamps as intervals, not points. Add tolerance to any comparison that crosses service boundaries.

For debugging: log both wall time and a logical sequence number. The wall time gives you human-readable context. The sequence number gives you causal ordering.

For architecture: don't design systems where correctness depends on two independent services agreeing on the current time. They won't.

Conclusion

Einstein wasn't being philosophical when he said there's no universal now.

He was being precise.

Two observers in different places, moving at different speeds, cannot agree on simultaneity. The universe doesn't provide a global clock. You have to define your reference frame and work within it.

Your distributed system operates under the same constraints.

The services don't share a clock. The network introduces asymmetric delays. "At the same time" doesn't mean anything unless you define it explicitly.

This isn't a solvable bug. It's the environment.

Design for it.

Timestamps tell you when a service thought something happened.
They do not tell you when it happened.
For that, you need causality, not clocks.

The Physics of Microservices: What Thermodynamics Teaches Us About System Design

Tasos Nikolaou — Tue, 30 Jun 2026 17:46:19 +0000

Thermodynamics is the study of energy, disorder, and the stubborn tendency of systems to fall apart.

Sound familiar?

I spent years studying physics before moving into software engineering. For a long time, I kept those two worlds separate. Then I started noticing something uncomfortable: the systems I was building kept obeying the same laws as the ones I used to model.

Not metaphorically. Structurally.

This article isn't a philosophical essay. It's a practical look at four thermodynamic laws and what they actually tell us about designing distributed systems.

The Problem: We Build Systems That Pretend Disorder Doesn't Exist

Most architecture diagrams look clean. Boxes. Arrows. Everything flowing in the right direction.

In reality:

Messages arrive out of order
Services fail silently
Caches diverge from the source of truth
Load spikes appear without warning
Retries amplify the problems they're trying to fix

We don't design for disorder. We design for the happy path, and then scramble when entropy shows up.

Thermodynamics has been studying disorder for 200 years. We should probably listen.

Law Zero: Equilibrium Is a Lie (But You Need a Reference Point)

The Zeroth Law says: if system A is in thermal equilibrium with system B, and B with C, then A and C are in equilibrium.

It sounds trivial, but it's not.

What it actually establishes is the concept of a shared reference. Temperature only means something because we agree on a scale. Without a common baseline, "hot" and "cold" are meaningless.

In distributed systems, this is the clock problem.

Two services can't coordinate if they don't agree on time. Or on schema versions. Or on what "success" means for a given operation.

The engineering lesson:

Before you design any inter-service protocol, define your zeroth law:

What is the shared reference for time? (NTP, logical clocks, vector clocks?)
What is the canonical schema, and who owns it?
What does "done" mean? (acknowledged, persisted, processed?)

Without a zeroth law, your services aren't communicating. They're guessing.

The First Law: Energy Is Conserved — So Is Load

The First Law: energy cannot be created or destroyed, only transformed.

In software terms: load doesn't disappear. It moves. This is the most violated intuition in backend engineering. You add a cache. Response times drop.

But the cache has to be invalidated. The invalidation adds write load. The write load adds latency elsewhere. The latency causes timeouts. The timeouts trigger retries. The retries spike your queue.

The load didn't go away. You transformed it!

Classic examples:

Rate limiting protects your service, but queues load upstream
Async processing moves latency from the user to the worker
CDN caching reduces origin load, but adds cache stampede risk on expiry
Circuit breakers shed load from a failing service, but pile it on callers who now need fallback paths

The engineering lesson:

When you add a layer that reduces pressure somewhere, ask: where did that pressure go?

Draw the full energy flow. Not just the happy path. The entire system. Load is conserved.

The Second Law: Entropy Always Increases

This is the one everyone knows, and the one engineers consistently underestimate.

The Second Law: in any closed system, entropy — disorder — always increases over time.

Left alone, systems don't stay ordered. They drift.

In distributed systems, this shows up constantly:

Configuration drift: services deployed weeks apart run different versions of shared config
Schema drift: a field added to one service's model doesn't propagate cleanly to consumers
State drift: replicas diverge from the primary under load
Behavioral drift: a service that worked fine in isolation behaves differently at scale

Entropy doesn't require mistakes. It's the default. You have to actively work against it.

The physics analogy is exact: maintaining order requires an energy input.

In software, that energy is:

Automated consistency checks
Schema registries with enforced contracts
Health monitoring with drift detection
Regular chaos testing to surface divergence before production does

The engineering lesson:

Your system will drift. The question is whether you find out first, or your users do.

Build in the entropy detection layer from the start. Don't treat consistency as the default. Treat it as something you have to earn, continuously.

The Third Law: You Can Never Reach Absolute Zero

The Third Law: as temperature approaches absolute zero, entropy approaches a minimum — but never reaches it.

You can never fully eliminate disorder. You can only reduce it. This is the law that most architecture documents ignore.

The fantasy: design a system with enough redundancy, enough retries, enough failover logic, and it will never fail.

The reality: you can push failure probability down. You cannot push it to zero.

What this means practically:

There is no perfect idempotency. You can get close. At-least-once delivery with deduplication is good. Exactly-once is a fiction that some message systems approximate, expensively.
There is no perfectly consistent distributed database. CAP theorem is just the Third Law wearing a different hat.
There is no zero-downtime deployment. There are deployments where downtime is small enough that users don't notice.

The engineers who build the most resilient systems aren't the ones who believe they can eliminate failure. They're the ones who build for graceful degradation — accepting that failure will happen and designing around it.

The engineering lesson:

Stop designing for zero failures. Design for bounded failures.

Define your acceptable entropy budget:

What is the acceptable message duplication rate?
What is the acceptable staleness window for cached data?
What is the maximum tolerable divergence between replicas?

Make these explicit. Then build your system to stay within those bounds, not to pretend the bounds don't exist.

Putting It Together: A Thermodynamic Checklist

When designing a new service or reviewing an existing architecture, I now ask four questions:

Zeroth Law — Do we have a shared reference?
Is time synchronized? Are schemas versioned and agreed upon? Does every service agree on what "success" means for each operation?

First Law — Where does the load go?
Every optimization moves load somewhere. Map the full flow. Find where it accumulates.

Second Law — What drifts?
Configuration, state, schemas, behavior. What's your detection mechanism? What's your recovery path?

Third Law — What's your entropy budget?
Define the failure modes you accept. Build your SLOs around reality, not around the fiction of zero failure.

Why This Matters More Now

Microservices amplify all of this.

A monolith has entropy problems. A microservices architecture has the same problems, multiplied by the number of service boundaries, and connected by network calls that can fail independently at each hop.

The Second Law doesn't care how good your orchestration is. The First Law doesn't care how well you've tuned your queues.

Physics doesn't negotiate.

The engineers who understand this don't fight against entropy. They build systems that stay stable despite it.

Conclusion

Thermodynamics didn't give me better debugging tools.

It gave me better questions.

Before worrying about which framework to use or which message broker to pick, ask: does this design respect the laws of the system it's running in?

Because at scale, your distributed system isn't just software.

It's a physical system.

And physical systems have rules.

The goal isn't to eliminate disorder.
The goal is to know exactly how much of it you can afford.

An LLM Walks Into General Relativity - Lessons from a Devoxx Talk

Tasos Nikolaou — Sun, 10 May 2026 17:18:00 +0000

Why fluent AI-generated technical content can still be fundamentally incorrect, and how to fix it with system design.

Introduction

At Devoxx, I presented a simple experiment:

What happens if you ask an LLM to generate an entire technical presentation on General Relativity?

The model produces something impressive:

well-structured slides
correct terminology
equations
citations
a coherent narrative

It looks like something you could present. And yet, parts of it are fundamentally wrong. Not obviously wrong, convincingly wrong.

This is the real problem with AI-generated technical content.

The Problem: Fluent ≠ Correct

Large Language Models are extremely good at:

structure
storytelling
pedagogy

But they are not built to preserve:

physical constraints
invariants
measurement consistency

In physics, that becomes "obvious" very quickly 😉.

Example:

A model might say:

"Light slows down in gravity, so time slows down."

This sounds reasonable. But it's wrong, or at best, deeply misleading, because:

locally, the speed of light is always c, constant
time dilation is defined through clock comparisons, not metaphors

This is what I call:

Frame confusion

The model mixes:

different observers
different measurement definitions
intuitive metaphors

...into a single explanation.

Everything reads smoothly. But the reasoning is broken.

Why Physics is a Perfect Stress Test

General Relativity is unforgiving.

You can't get away with:

vague explanations
metaphor-only reasoning
mixing frames of reference

Every statement must answer:

"How would you measure that?"

If you can't answer that, the explanation is incomplete, or wrong. This makes physics an ideal domain to expose LLM weaknesses.

From Prompting to System Design

Instead of trying to "prompt better", I built a system around the model.

The goal:

Not to make the model smarter, but to make the output auditable and correctable.

Architecture Overview

The system is a multi-agent pipeline:

Sources → Chunking 
        → Retrieval (RAG)        
        → Author Agent (generate slides)
        → Schema Validation
        → Post-processing
        → Physics Rule Engine
        → Critic Agent
        → Refinement Loop
        → PowerPoint Rendering

Key Components

1. Structured Generation

The model doesn't output free text. It must generate strict JSON:

slide types
bullet constraints
equations
citations

Validated with Pydantic. If it doesn't parse, it doesn't ship.

2. Deterministic Validation (The "Physics Linter")

I implemented rule-based checks like:

Time dilation must reference clocks or measurements
Gravitational waves must reference strain or detectors
No "black holes suck everything in" explanations
Distinguish event horizon vs singularity

These rules catch systematic failure patterns instantly.

3. Critic Agent

A second LLM reviews the output:

checks clarity
checks reasoning
suggests corrections

But importantly, it runs after deterministic validation

4. Refinement Loop

Generate → Validate → Critique → Revise

This loop runs until:

errors are reduced
or a maximum number of iterations is reached

Results

From a real run:

Draft deck → 6 failing slides
After refinement → 4 failing slides

We didn't achieve perfection. But we achieved something more important:

We made correctness measurable.

What Still Fails

Even with this pipeline:

Citations may look correct but not truly support claims
Subtle reasoning errors remain
Frame confusion is hard to eliminate
Models can satisfy rules while staying vague

Human review is still necessary.

Key Insight

Reliable AI is not a prompting problem.
It's a system design problem.

What You Can Reuse

These patterns generalize beyond physics:

legal documents
financial reports
medical summaries
architecture decisions

Use:

structured outputs
deterministic validation
domain-specific rules
critique loops

The project

GitHub Repo: https://github.com/tase-nikol/gr-deck-agent

Example commands:

make index
make draft
make review
make refine
make replay

Devoxx Talk

You can watch the full talk here:

YouTube: https://www.youtube.com/watch?v=NanGs7ZMQEE

Final Thought

LLMs don't "understand" systems.

They generate plausible descriptions of them.

If your domain has constraints, invariants, or correctness requirements:

You need to build those constraints into the system,not hope the model learns them.

Conclusion

If you're working with AI-generated technical content, I'd love to hear:

what failure modes you've seen
how you validate outputs
what worked (or didn't)

AI is fluent
But reality is not optional

When Chrome Ate My RAM: Designing a Pressure-Aware Tab Orchestrator with Rust

Tasos Nikolaou — Wed, 01 Apr 2026 16:30:07 +0000

Chrome wasn't "crashing."

It was just...slowly suffocating my system.

Over time, RAM usage would creep up. Background tabs accumulated state. Other applications started freezing. The fan would spin up. And yet, nothing looked obviously wrong. No single tab was the culprit.

The problem wasn't too many tabs.
The problem was a lack of coordination between the browser and the system.
So I built something to experiment with that idea.

This article explains the architecture and reasoning behind a hybrid Chrome extension & Rust native host that manages tab lifecycle based on real system pressure and user context.

The Problem: Browser Entropy

Modern browsers are operating systems.

They manage:

Dozens of isolated processes
Background timers
Network activity
Memory-heavy applications (Jira, GitHub, Gmail, ChatGPT, Claude 😊 etc.)

Most tab suspension tools rely on a simple rule:

"If a tab hasn't been used in X minutes, suspend it."

That's convenient, but blind.

They don't know:

Whether the system is under memory pressure
Whether CPU is spiking
Whether you're on battery
Whether the tab is part of your active workflow

They operate on time, not state.

What I wanted was:

A deterministic, pressure-aware, context-sensitive lifecycle engine.

Not AI. Not cloud analytics. Just a well-structured system.

Design Goals

Before writing any code, I defined constraints:

Deterministic behavior (no black-box magic)
No cloud, no telemetry
Respect user intent (never discard active or pinned tabs)
Pressure-aware decisions
Context-aware heuristics
Clean separation of responsibilities

This last one became the most important architectural decision.

Architecture Overview

The system consists of two components:

Chrome Extension (MV3)
  - Tab activity tracking
  - Focus clustering
  - TTL gating & guardrails
        ↓ Native Messaging
Rust Native Host
  - System metrics (RAM, CPU, Battery)
  - Pressure scoring engine
  - Deterministic classification

Why split it?

Chrome extensions cannot access low-level system metrics like real memory pressure in a reliable way.

So I separated concerns:

The extension manages browser lifecycle.
The Rust native host understands system state.

They communicate through Chrome's Native Messaging API.

This keeps the system clean:

Browser logic stays in the browser.
System logic stays native.

The Pressure Engine (Rust)

Instead of checking raw RAM percentage, I built a weighted pressure scoring model.

The Rust host collects:

Total RAM
Used RAM
Free RAM
CPU usage
Battery state (if available)

From these, it computes:

pressure_score (0-100)
pressure_level (LOW / MEDIUM / HIGH)
pressure_reasons (RAM_HIGH, CPU_ELEVATED, ON_BATTERY, etc.)

RAM is the dominant signal.
CPU acts as a modifier.
Battery adds a small aggressiveness bias.

The goal is not to be perfect, it's to be consistent and explainable.

Instead of saying:

"System busy."

It says:

HIGH pressure because RAM_HIGH + ON_BATTERY.

That reason tagging matters for transparency.

Context Awareness - Focus Clustering

Not all inactive tabs are equal. A tab opened 30 minutes ago in your active workflow is very different from a forgotten tab in another window.

So I introduced Focus Mode.

Focus clustering is based on:

Same hostname as active tab
Recent activity window
Same window constraint
Cluster size cap

Tabs inside the active "cluster" use a longer TTL.Tabs outside the cluster expire faster under pressure. This makes the system:

Less disruptive
More aligned with user context
Less likely to discard something you'll immediately need

It's still deterministic, but just smarter.

Guardrails & Safety

Aggressive resource management can easily become destructive. So, strict guardrails were built in:

Never discard active tabs
Never discard pinned tabs
Never discard audible tabs
Respect protected domains
Enforce TTL minimums
Apply cooldown between prune cycles

This prevents oscillation and surprise behavior. The goal is not maximum efficiency. The goal is controlled stability.

Why Rust?

Rust was chosen for the native host because:

Memory safety
Explicit modeling
Strong type system
Clean modular architecture
Lightweight binary

The Rust side is structured into modules:

metrics, system state collection
battery, optional battery signal
pressure, scoring logic
protocol, native messaging transport
state, API contract

This makes the native host feel like a real subsystem, not a script.

What It Achieves

In practice, this system:

Reduces RAM pressure under load
Keeps active workflows intact
Makes browser behavior predictable
Avoids blind "time-based" suspension
Plays nicer with other system applications

It doesn't eliminate memory usage. It orchestrates it.

What I Learned

A few things stood out during this project:

1. MV3 Service Workers Have Quirks

Extension background scripts are ephemeral. State management must be deliberate.

2. Determinism Beats "Smartness"

Clear, explainable rules feel safer than opaque heuristics.

3. Separation of Concerns Changes Everything

Keeping system logic in Rust and browser logic in the extension made experimentation much easier.

4. Observability Matters

Reason tagging and structured logging made debugging and tuning far easier.

Future Directions

This project is still evolving. Some experimental directions:

Event-driven pressure signals (instead of polling)
Chrome process memory integration
Predictive return probability modeling
Offline data analysis of tab lifecycle patterns
Adaptive TTL tuning

The architecture supports these without becoming tangled.

That was intentional.

Conclusion

Chrome didn't have a bug. It was just operating without coordination. By introducing a pressure-aware, context-sensitive orchestration layer, the browser becomes less chaotic and more cooperative with the system.

This project started as frustration with RAM usage. It turned into an exploration of how browsers and operating systems can communicate more intelligently, without AI hype, and without cloud dependencies.

Just clean architecture and deterministic policy!

Checkout the project here:
Github: https://github.com/tase-nikol/tab-memory-orchestrator

Sometimes the problem isn’t that a system is broken
It’s that its parts aren’t talking to each other.

PromptCache Part II: When High Cache Hit Rates Become Dangerous

Tasos Nikolaou — Thu, 05 Mar 2026 07:50:27 +0000

A benchmark-driven look at semantic cache safety and intent isolation.

In the previous article, "Stop Paying Twice for the Same LLM Answer" - I introduced PromptCache as a semantic caching layer designed to reduce LLM cost and latency.

The premise was simple, if two prompts are semantically similar, we shouldn't pay for the answer twice. The results were compelling: high cache hit rates, significant cost reduction, lower latency. But after deploying and stress-testing the design, a more important question emerged:

What guarantees that a cache hit is actually correct?

Reducing cost is easy.
Ensuring safe reuse is harder.

This article documents the experiment that reshaped PromptCache's architecture, and why intent isolation became a non-negotiable design constraint.

Measuring Semantic Cache Safety in LLM Systems

Most semantic caches follow a simple pattern:

Embed the prompt
Retrieve the nearest cached embedding
If similarity ≥ threshold then reuse the answer

This works well for performance.
But it assumes something that isn't guaranteed:

That semantic similarity implies safe reuse.

To test that assumption, I built a controlled benchmark.

The Question

Can cosine similarity thresholding alone guarantee safe reuse? Or do we need structural isolation between tasks? To answer this, I defined a metric:

Unsafe Hit

A cache hit is unsafe if the returned answer belongs to a different task (intent) than the incoming request. This measures semantic collision, not embedding quality.

What Is Intent Isolation?

Intent isolation means:

Partition the semantic cache by task boundary before performing similarity search.

Instead of searching across all cached entries, we search only within the matching task. Similarity becomes a refinement step, not a boundary mechanism.

Figure 0 - Semantic Cache Search Space.
Without isolation, similarity search spans all tasks in one shared embedding space.

With isolation, search is restricted to the matching intent partition.

Experimental Setup

I evaluated semantic caching across:

Embedding model: all-MiniLM-L6-v2
Backends:
- In-memory brute-force cosine
- Redis (HNSW via RediSearch)
Workloads:
- Support queries
- RAG-style retrieval questions
- Creative prompts
Threshold sweep: 0.82 -> 0.92
~2400 requests per configuration

Two configurations were tested:

1. No Intent Isolation

All prompts shared the same semantic space.

2. Intent Isolation Enabled

Cache entries were partitioned by intent_id.

Each configuration was evaluated across identical prompt sequences to ensure comparability.
Unsafe hits were computed by comparing stored intent_id against request intent.

Result 1 - Hit Rate Looked Excellent

Without intent isolation:

Hit rate: ~97-99% With intent isolation:
Hit rate: 13-38% depending on threshold

Figure 1 - Hit Rate vs Threshold. Without intent isolation, semantic caching achieves ~98% hit rate. Enabling intent partitioning significantly reduces reuse density.

At first glance, the non-isolated configuration looks superior. But this metric is incomplete.

Result 2 - Unsafe Hit Rate Reveals the Problem

Without intent isolation:

Unsafe hit rate: ~95-100% With intent isolation:
Unsafe hit rate: 0%

Figure 2 - Unsafe Hit Rate vs Threshold. Similarity thresholding alone does not prevent cross-intent reuse. Nearly all cache hits become unsafe without intent partitioning.

This pattern was consistent across support, RAG, and creative workloads. In other words:

Almost every "successful" cache hit without isolation was incorrect.

This is not a marginal effect. It is structural cross-contamination.

Why Similarity Is Not Enough

Embedding similarity measures geometric proximity in vector space.
Intent boundaries are categorical.

Cosine similarity answers:

"Are these prompts semantically related?"

It does not answer:

"Are these prompts operationally interchangeable?"

Semantic closeness is continuous and task equivalence is discrete. Threshold tuning cannot convert a continuous metric into a categorical guarantee, but, partitioning can.

Result 3 - Backend Did Not Affect Correctness

Both Redis (HNSW) and the in-memory backend produced identical:

Hit rate curves
Unsafe hit rate curves

This was expected, due to the fact that both implemented cosine nearest-neighbor search with identical threshold logic. Correctness was dominated by key structure, not vector store implementation.

Backend choice affects:

Persistence
Multi-process access
Scalability
Latency under load

But correctness properties should not depend on storage details!

Cost Savings Followed Hit Rate

In this benchmark, each miss triggered a full LLM call with similar token usage.

As a result, cost_savings ≈ hit_rate, which confirms internal consistency. But cost reduction is meaningless if reuse is unsafe, meaning that correctness precedes optimization.

Production Implications

If you rely solely on similarity thresholding:

You will inflate hit rates
You will inflate cost savings
You may silently reuse incorrect answers

This is particularly dangerous in:

Multi-tenant systems
Support bots
RAG pipelines
Tool-driven workflows

The correct architectural pattern is:

Partition first.
Threshold second.

Similarity is a refinement mechanism, not a safety boundary.

Limitations

This was a controlled benchmark.

Dataset size was modest (~2-3k prompts)
Workloads were synthetic but structured
Extreme-scale recall behavior was not evaluated
Concurrency stress was not measured

The goal was to isolate semantic collision behavior, not benchmark vector database scalability.

Future work should explore:

Larger datasets
Cross-model embedding drift
Concurrency stress testing
Partial response reuse

Core Insight

The dominant factor in semantic cache correctness is not:

The embedding model
The vector database
The similarity threshold

It is key design.
Intent isolation is not an optimization.
It is a safety requirement.

Final Takeaway

A 98% cache hit rate looks impressive.
But without structural boundaries, it may be misleading.

If your semantic cache shows:

98% hit rate
98% cost savings

Ask yourself, how many of those hits are actually correct?
Optimization without isolation is probabilistic reuse. If you're building LLM infrastructure, this is not an academic nuance, but a production concern.

Similarity optimizes reuse.
Isolation guarantees correctness.

PromptCache Part I: Stop Paying Twice for the Same LLM Answer

Tasos Nikolaou — Tue, 24 Feb 2026 08:23:32 +0000

Designing a semantic cache layer for cost and latency optimization in LLM systems.

Most LLM cost isn’t spent on novelty.
It’s spent on repetition, requests that are semantically identical, but syntactically different.

PromptCache was built to eliminate that redundancy.

The Invisible Cost Leak in LLM Systems

If you’re running an LLM in production, you are almost certainly paying for this:

"How do I reset my password?"
"I forgot my password, what do I do?"
"Steps to reset account password?"
"Help me change password"

Different strings.
Same intent.
Same answer.
Different billable request.

Traditional caching doesn't help because:

"How do I reset my password?" != "Steps to reset account password?"

Exact match fails.
But meaning hasn't changed.
That's where semantic caching comes in.

The Theory: Why This Works

LLMs don't understand text as strings.
They convert text into vectors (embeddings).
Two sentences with similar meaning produce vectors that are close together in high-dimensional space.

Example (simplified):

"Reset my password"
      ↓
[0.12, -0.87, 0.44, ...]

"How do I change my password?"
      ↓
[0.11, -0.89, 0.41, ...]

These vectors are very close.

So instead of asking:

"Have I seen this exact string before?"

We ask:

"Have I seen something semantically similar before?"

If the similarity is high enough, we reuse the answer.
That's semantic caching.

How It Works in Practice

When a request comes in:

User Prompt
     ↓
Embedding
     ↓
Vector search in Redis
     ↓
High similarity?
     ↓
Yes → Return cached response
No  → Call LLM and store result

You're adding a semantic memoization layer in front of your LLM.

Real Results

In a support-heavy workload with repetitive queries:

~60% cache hit rate
~50% reduction in token usage
~40% lower API spend

Results vary by workload density and repetition patterns, but in structured environments, the impact is immediate.

Example Implementation

Here's a simplified example using Redis vector search:

from promptcache import SemanticCache
from promptcache.backends.redis_vector import RedisVectorBackend
from promptcache.embedders.openai import OpenAIEmbedder
from promptcache.types import CacheMeta

embedder = OpenAIEmbedder(model="text-embedding-3-small")

backend = RedisVectorBackend(
    url="redis://localhost:6379/0",
    dim=embedder.dim,
)

cache = SemanticCache(
    backend=backend,
    embedder=embedder,
    namespace="support-bot",
    threshold=0.92,
)

meta = CacheMeta(
    model="gpt-4.1-mini",
    system_prompt="You are a helpful support assistant.",
)

result = cache.get_or_set(
    prompt="How can I change my password?",
    llm_call=my_llm_call,
    extract_text=lambda r: r.output_text,
    meta=meta,
)

print(result.cache_hit)

That's it.

No orchestration framework required.

If you want to try this approach, I packaged it up here:

GitHub: https://github.com/tase-nikol/promptcache
PyPI: https://pypi.org/project/promptcache-ai/

Install:

pip install promptcache-ai

When This Works Best

Semantic caching is powerful when:

Prompts are repetitive
Temperature is low
Answers are stable
Volume is high

It won't help much for:

Highly personalized prompts
Creative writing
Rapidly changing context

In those cases, novelty dominates repetition, and caching provides diminishing returns.

The Bigger Insight

Most LLM systems are fundamentally stateless.
They recompute answers even when nothing meaningful has changed.

Semantic caching introduces selective memory, reusing intelligence only when it is economically justified.

Instead of optimizing prompts endlessly, sometimes the smarter move is optimizing infrastructure.

If you're building LLM systems in production, semantic caching is one of the highest-leverage optimizations you can add.

But optimizing cost raised a more uncomfortable question:
What guarantees that a cache hit is actually correct?

In the next article, we examine how high hit rates can silently mask semantic errors, and why PromptCache evolved beyond threshold tuning.

Intelligence is expensive.
Memory is cheap.
Use both wisely.

Building a Framework-Agnostic Health Check Library for Python Microservices

Tasos Nikolaou — Tue, 17 Feb 2026 21:31:51 +0000

From duplicated /health endpoints to a published PyPI package - an engineering deep dive.

The Problem: Death by Copy-Paste Health Checks

In a typical microservice architecture, health endpoints start simple:

GET /health
GET /ready

But over time, reality sets in.

Some services use:

Django + PostgreSQL + Redis + Celery
FastAPI + SQLAlchemy + Redis
BFFs that depend on upstream HTTP services
RabbitMQ + background workers
Async stacks mixed with sync stacks

Each service needs:

Liveness checks
Readiness checks
Dependency verification
Timeouts
Structured JSON output
Correct HTTP status codes

And before long, every service has its own slightly different HealthService.

Different thresholds.
Different response formats.
Different timeout logic.
Different readiness semantics.

That's when I realized:

Health checks are infrastructure. They should not be rewritten per service.

So I built PulseCheck - a framework-agnostic health and readiness library for Python.

Design Goals

Before writing a single line of code, I defined constraints:

Framework-agnostic core
Pluggable dependency checks
Async-first design (to support FastAPI)
Sync compatibility (for Django)
No forced dependency pollution
Kubernetes-friendly readiness semantics
Optional dependency extras
Clean, structured JSON output
Production-safe timeouts

This wasn't just about code reuse.

It was about architectural consistency.

Architecture: Core + Adapters

The key design decision was separation of concerns.

pulsecheck/
│
├── core/        ← Framework-agnostic health engine
├── fastapi/     ← FastAPI adapter
└── django/      ← Django adapter

1. Core Engine

The core layer contains:

Health registry
Health aggregation logic
Status combination rules
Dependency check base class
Timeout handling
Response schema

It has zero framework dependencies.

The core doesn't know what FastAPI or Django is.

2. Pluggable Checks

Each dependency is implemented as a check:

SQLAlchemyAsyncCheck
DjangoDBCheck
RedisAsyncCheck
RedisSyncCheck
RabbitMQKombuCheck
CeleryInspectCheck
HttpDependencyCheck

Each check:

Has a name
Has a timeout
Has a degraded threshold
Returns structured results

Example:

registry = HealthRegistry(environment="prod")

registry.register(SQLAlchemyAsyncCheck(engine))
registry.register(RedisAsyncCheck(redis_url))
registry.register(CeleryInspectCheck(celery_app))

No monolithic service class.
Just composition.

Async-First, Sync-Compatible

FastAPI is async.
Django is traditionally sync.

Instead of creating two engines, the core is async-first.

Sync checks are wrapped using:

asyncio.to_thread(...)

This gives:

Async compatibility
Non-blocking readiness
Unified aggregation logic

This avoids duplicating the health engine.

Readiness vs Liveness

This is often misunderstood.

Liveness:

"Is the process alive?"

Readiness:

"Can this service safely receive traffic?"

PulseCheck separates them cleanly:

registry.liveness()
await registry.readiness()

Readiness runs dependency checks.
Liveness does not.

This mirrors Kubernetes probe behavior.

Handling Degraded States

Health isn't binary.

Instead of just UP or DOWN, PulseCheck supports:

HEALTHY
DEGRADED
UNHEALTHY

If a dependency is slow but responding:

{
  "status": "DEGRADED",
  "response_time_ms": 750
}

This gives operational insight without triggering restarts.

Optional Dependencies Done Right

One of the most important design decisions was dependency management.

FastAPI projects already have FastAPI.
Django projects already have Django.

The library must not force unnecessary installations.

In pyproject.toml:

[project.optional-dependencies]
fastapi = ["fastapi>=0.100"]
django = ["Django>=4.2"]
redis_async = ["redis>=5.0"]
rabbitmq = ["kombu>=5.3"]
celery = ["celery>=5.3"]

Now:

FastAPI service:

pip install pulsecheck-py[fastapi,redis_async]

Django service:

pip install pulsecheck-py[django,redis_sync]

Clean. Explicit. Controlled.

Hiding Health Endpoints From Swagger

Health endpoints are infrastructure endpoints.

In FastAPI:

@router.get("/health", include_in_schema=False)

They exist.
They work.
They don't pollute public API docs.

Small detail. Big professionalism signal.

Testing Before Publishing

Before uploading to PyPI, I tested:

Editable installs (pip install -e .)
Wheel builds (python -m build)
Installation from built wheel
Installation from TestPyPI
Optional extras resolution
Fresh virtual environments

I also learned something important:

TestPyPI contains junk packages that can interfere with dependency resolution. Always use:

--extra-index-url https://test.pypi.org/simple/

Not --index-url.

Small ecosystem lesson.

Publishing to PyPI

Publishing was straightforward:

python -m build
python -m twine upload dist/*

Important rule:

You cannot overwrite a version on PyPI.

Every change requires a version bump.

This enforces discipline.

Engineering Lessons Learned

Design the API before writing implementation.
Keep core logic framework-agnostic.
Async-first design avoids duplication.
Optional dependencies prevent ecosystem pollution.
Health endpoints are infrastructure, not business logic.
Packaging and versioning require discipline.
Publishing is easier than maintaining.

Why This Matters

Microservices suffer from invisible duplication.

Health checks are often treated as boilerplate.

But consistency in infrastructure code improves:

Operational clarity
Monitoring integration
Kubernetes reliability
Onboarding speed
Codebase maintainability

PulseCheck turned copy-paste health logic into a reusable, composable abstraction.

Future Roadmap

OpenTelemetry hooks
Prometheus integration
Circuit-breaker awareness
Startup probe support
Health history tracking
Async worker health strategies

Final Thoughts

Publishing a library is not about writing code.

It's about:

API design
Dependency discipline
Versioning strategy
Documentation clarity
Ecosystem compatibility

PulseCheck started as internal cleanup.
It became a reusable infrastructure layer.

If you're duplicating health logic across services, consider abstracting it.

Your future self will thank you.

Links

PyPI: https://pypi.org/project/pulsecheck-py/
GitHub: https://github.com/tase-nikol/pulsecheck-py

If you'd like feedback on the architecture or want to contribute, feel free to reach out.

"PulseCheck is intentionally minimal today. But its architecture allows deeper observability and resilience integrations"