DEV Community

Cover image for Designing a Distributed Cache System
Nguyen Nguyen
Nguyen Nguyen

Posted on

Designing a Distributed Cache System

#systemdesign

1.Overview

A distributed cache stores frequently accessed data in memory across multiple servers to reduce database load and improve response latency.

Typical use cases:

Session storage
User profile caching
Product catalog
API response cache
Configuration data
Leaderboards
Rate limiting

2.Requirements

Functional 

GET(key)
SET(key, value, TTL)
DELETE(key)
EXPIRE(key)
Batch operations (optional)
Enter fullscreen mode Exit fullscreen mode

Non-functional 


< 2 ms read latency
Millions of QPS
Horizontal scaling
Fault tolerance
High availability
Data expiration
Memory efficient
Enter fullscreen mode Exit fullscreen mode

3. High Level Architecture 

                Client

                  |

        +------------------+
        | Cache Client SDK |
        +------------------+

                  |

         Consistent Hashing

      +--------+--------+--------+
      | Node A | Node B | Node C |
      +--------+--------+--------+

          Replication

      +--------+--------+--------+
      |Backup A|Backup B|Backup C|
      +--------+--------+--------+
Enter fullscreen mode Exit fullscreen mode

4.Components

a.Cache Client

Responsible for

hashing key
locating node
retry
connection pooling
serialization
Enter fullscreen mode Exit fullscreen mode

Client-side routing avoids an extra proxy hop.

b.Cache Nodes

Each node stores

HashMap<Key, Entry>

Entry

struct Entry {
    Value value
    Timestamp expireAt
}
Enter fullscreen mode Exit fullscreen mode

c.Metadata Service

Stores

cluster topology
node health
hash ring
replica mapping
Enter fullscreen mode Exit fullscreen mode

Can use

etcd
ZooKeeper
Consul
Enter fullscreen mode Exit fullscreen mode

Monitoring 


Hit Rate
Miss Rate
Memory Usage
QPS
Evictions
Replication Lag
Enter fullscreen mode Exit fullscreen mode

5.Partitioning

Consistent Hashing 

Ring

NodeA

NodeB

NodeC

key -> nearest clockwise node
Enter fullscreen mode Exit fullscreen mode

Benefits

  • only small percentage of keys move when new node added
  • horizontal scaling
  • balanced distribution

6.Cache Eviction

Memory is limited.

Need eviction policy.

LRU

Least Recently Used

LFU

Least Frequently Used

TTL

Each key has: expireAt

Two methods.

Passive expiration


GET

↓

expired?

↓

delete
Enter fullscreen mode Exit fullscreen mode

Cheap.

Active expiration


Background thread scans expired keys.

Every second

↓

scan

↓

remove
Enter fullscreen mode Exit fullscreen mode

Redis combines both.

7.Cache Consistency

Cache Aside

Most common.

Read


Cache

↓

Miss

↓

Database

↓

Cache

↓

Return
Enter fullscreen mode Exit fullscreen mode

Write


DB

↓

Delete Cache
Enter fullscreen mode Exit fullscreen mode

Pros

Simple.

Cons

Temporary stale data.

Read Through

App

↓

Cache

↓

Database
Enter fullscreen mode Exit fullscreen mode

Application never accesses DB directly.

Write Through

App

↓

Cache

↓

Database
Enter fullscreen mode Exit fullscreen mode

Strong consistency.

Higher latency.

Write Behind

App

↓

Cache

↓

ACK

↓

Flush DB later
Enter fullscreen mode Exit fullscreen mode

Fast.

Risk of data loss.

Top comments (0)