Previously in this series, we looked at why caching exists. It can reduce latency, lower database load, and help an application handle more traffic.
But there is a natural next question:
What actually happens inside a cache?
To answer that, we need to understand where the data lives, how we find it, how long it stays there, what happens when the cache fills up, and why location matters.
Once these concepts click, technologies such as Redis, Memcached, browser caches, CDNs, and application-level caches become much easier to reason about.
Memory vs. disk: why is a cache so fast?
When people say a cache is fast, they are often talking about memory — specifically RAM.
A database may use memory too. Modern databases aggressively cache data in RAM. The point is not that databases are always slow or that caches are always in RAM. The point is that a purpose-built cache often keeps frequently used data and the lookup path extremely simple.
Memory is fast, but it is limited and usually treated as less durable than disk storage. Disk gives us much more capacity and persistence, but accessing data from disk or through a more complex storage path generally costs more time.
Cache → optimise for fast access
Database → optimise for durable, structured, reliable storage
This is a mental model, not an absolute rule. Modern systems blur the boundary: databases use RAM caches, operating systems cache disk pages, and some caches can persist data.
The key-value model
Most caching systems become easier to understand when you think of them as a giant dictionary.
key: "user:123"
value: { "name": "John", "country": "USA" }
The application knows the key. The cache uses that key to find the value quickly.
Good cache keys are predictable and unique enough to avoid collisions:
user:123
product:987
product:987:details
recommendations:user:123
Key design becomes increasingly important as systems grow. Poorly designed keys can cause collisions, make invalidation difficult, or create unexpected hot spots.
TTL: how long should cached data live?
A cache cannot keep every value forever. At some point, cached data needs to expire. That is where TTL — Time To Live — comes in.
weather:atlanta → { ... } TTL = 5 minutes
For the next five minutes, requests can use the cached value. After the TTL expires, the entry is considered expired and the application may need to fetch fresh data.
TTL is a trade-off. A very short TTL keeps data fresh but causes more cache misses. A very long TTL improves cache efficiency but increases the chance of serving stale information.
There is no magic TTL. A stock price may need a very different freshness strategy from a country-code lookup table. A product description may tolerate minutes or hours of staleness, while an account balance may require much stricter rules.
The right TTL comes from the business requirement — not from a universal caching rule.
What happens when the cache is full?
Imagine your cache has a fixed amount of memory and every new item wants a place. Eventually, there is no room.
The cache needs a way to decide which existing item should be removed. This is called an eviction strategy.
Think of it like a small desk. When the desk is full and a new document arrives, you need a rule for deciding which old document gets thrown away.
| Policy | Full name | What it looks at |
|---|---|---|
| LRU | Least Recently Used | Recency — how long since this was last touched |
| LFU | Least Frequently Used | Frequency — how often this has been accessed |
| FIFO | First In, First Out | Insertion order — how long this has been here |
The distinction sounds academic until you apply all three to the same cache at the same moment. Consider a cache that is full at three entries when a fourth item arrives:
Same data, same instant, three different victims. FIFO throws out the entry that has been accessed the most, because it happened to arrive first. LFU throws out the entry that was just used a minute ago, because it hasn't accumulated many accesses yet.
Neither is a bug. Each policy encodes an assumption about which data you are likely to need next, and that assumption is either a good fit for your access pattern or a bad one. Real systems may offer additional policies or combinations.
The important principle is this: when capacity is limited, the cache needs a policy for deciding what stays.
Cache locality: keep data close
Another important idea is locality.
Imagine an application server in Atlanta accessing a nearby cache versus repeatedly making requests to a cache across the country. Both may work, but the network path matters.
Application → nearby cache → response
Application → distant cache → network → response
Every network hop introduces latency and another possible failure point.
This is why system designers often try to keep frequently accessed data physically or logically close to the application that needs it. The same idea appears at many layers:
- CPU caches keep data close to the processor
- Browser caches keep content close to the user
- CDNs keep content close to geographic regions
- Application caches keep frequently used data close to the application
Locality is not only geography. It can also mean reducing unnecessary network calls and expensive processing.
Understanding the performance numbers
Saying "the cache is fast" is not enough. You need to understand what fast actually means.
Latency
Latency measures how long an individual operation takes.
Cache lookup: 2 ms
Database query: 80 ms
These are illustrative numbers, not universal benchmarks. Real measurements depend on hardware, network distance, workload, serialization, concurrency, and many other factors.
Throughput
Throughput is how much work the system can handle over time — for example, requests per second.
5,000 cache operations/sec
A cache with low latency but insufficient throughput can still become a bottleneck.
Cache hit ratio
The hit ratio tells us how often requested data is found in the cache.
1,000 requests
900 hits
100 misses
Hit ratio = 90%
A high hit ratio is generally useful because more requests avoid the backend. But it is not a universal score of success.
Imagine a cache with a 99% hit ratio where each cache lookup is expensive, or where the 1% of misses trigger extremely costly database operations. The hit ratio alone does not tell the whole story.
Look at the numbers together
latency + throughput + hit ratio + memory usage + backend load
For example, after introducing a cache, API latency might drop from 150 ms to 40 ms and database CPU might fall significantly — but cache memory could be growing too quickly. That is not simply success or failure; it is a signal for the next design decision.
Putting it all together
Let's walk through one request:
- User requests product 987
- Application builds the key:
product:987 - Cache checks the key
- Hit? Return the cached value
- Miss? Query the database
- Store the result in the cache
- Apply a TTL
- If the cache becomes full, an eviction policy decides what leaves
Caching is no longer just "put data somewhere fast." It is a small system with rules:
- Where is the data?
- How do I find it?
- How long does it live?
- What happens when it expires?
- What happens when there is no space?
- How fast is the lookup?
- What happens when the cache fails?
Those questions are the foundation of good cache design.
Caching is a layer, not just a storage box
One useful mental model is to think of caching as a layer between the application and the source of truth.
User
↓
Application
↓
Cache ← fast, temporary copy
↓
Database ← durable source of truth
The cache exists to reduce the cost of repeatedly reaching the source of truth.
That distinction becomes very important when we start discussing consistency, invalidation, and failure handling.
What's next
So far, we have looked at what caching solves and how a basic cache works.
The next question is architectural: where should the cache live?
Should it live inside the application? On a separate server? At the edge? In the browser? In front of a database?
Next in this series, we'll explore the different places caching can exist — from browser caches and CDNs to application and distributed caches — and understand why choosing the right layer can be just as important as choosing the caching technology.
Which eviction policy does your cache actually use — and do you know why that one? I'd be curious to hear in the comments whether it was a deliberate choice or the default that shipped with the tool.
Top comments (0)