Why I Built Another In-Memory Cache for Go

There are already several good in-memory cache libraries for Go.

So when I started building pacecache, the obvious question was:

Why another one?

I wasn't trying to build a cache that would be universally better than every existing alternative. Cache design is a collection of trade-offs: lock contention, eviction quality, capacity utilization, expiration, memory overhead, and implementation complexity all pull in different directions.

What I wanted was a relatively small, generic in-process cache with explicit behavior around bounded capacity, expiration, cache-aside loading, and concurrent mutations.

The map part was straightforward.

The interesting part was deciding what should happen when all of those things interact concurrently.

Start with bounded capacity

An in-process cache should have a clear limit.

In pacecache, that limit is an entry budget, not a byte-size memory limit. By default, a cache can hold up to 10,000 entries, uses one storage segment, and has no time-based expiration.

A cache with a larger capacity and a default TTL looks like this:

cache, err := pacecache.New[string, User](
    pacecache.WithMaxEntries(100_000),
    pacecache.WithTTL(5*time.Minute),
)
if err != nil {
    return err
}

Eviction uses exact LRU ordering within each storage segment.

With one segment, the entire cache shares one LRU and one capacity budget. For workloads with heavier concurrency, storage can be split into independent segments:

cache, err := pacecache.New[string, User](
    pacecache.WithMaxEntries(100_000),
    pacecache.WithSegmentCount(64),
)

The total entry budget is divided across those segments, and their capacities add up to the configured maximum. Each segment owns its own storage, LRU list, expiration index, and lock.

That makes segmentation a trade-off rather than a free performance switch.

More segments can reduce contention because unrelated keys are more likely to hit different locks. But each segment also enforces its own local capacity. If keys are distributed unevenly, one segment may start evicting while another still has unused space.

That's why pacecache defaults to one segment rather than choosing a large number automatically.

The right segment count depends on the workload. It's something worth measuring rather than guessing.

Expiration and cleanup are different problems

TTL sounds simple until expiration enters the hot path.

One distinction I wanted to preserve is:

An entry being expired is not the same thing as that entry already being physically removed from storage.

Once an entry reaches its deadline, lookup paths no longer treat it as live. An expired entry encountered during a lookup is treated as a miss and removed.

Physical reclamation can also happen explicitly or through optional background cleanup.

The default TTL can be configured together with jitter:

cache, err := pacecache.New[string, User](
    pacecache.WithTTL(5*time.Minute),
    pacecache.WithJitter(30*time.Second),
)

Jitter adds a random duration below the configured limit when an expiring entry is stored. This helps spread expiration deadlines instead of letting a large group of entries expire at exactly the same time.

Individual writes can choose their own expiration policy:

cache.Set("user:1", user, pacecache.DefaultExpiration)
cache.Set("user:2", user, pacecache.NoExpiration)
cache.Set("user:3", user, 30*time.Second)

Sliding expiration is optional as well.

When enabled, a successful read of a live expiring entry refreshes its deadline using the effective TTL selected when that entry was stored. If jitter was applied, that effective TTL is reused rather than randomized again on every read.

The important part is that TTL correctness does not depend on a cleanup goroutine.

Background cleanup is useful for reclaiming expired entries that nobody touches again, but it isn't what decides whether an entry is logically alive.

I prefer that separation because scheduling cleanup and enforcing expiration are two different concerns.

Cache-aside loading introduces coordination

The next problem appears when a cache is used in a cache-aside pattern.

Imagine several goroutines requesting the same missing key at roughly the same time. Without coordination, all of them can perform the same database query or remote request.

pacecache coalesces concurrent misses for the same key into one shared load.

A per-call loader can be supplied through GetOrLoadFunc:

user, found, err := cache.GetOrLoadFunc(
    ctx,
    userID,
    func(ctx context.Context, key string) (User, bool, error) {
        return findUser(ctx, key)
    },
)

The loader contract is:

func(ctx context.Context, key K) (value V, found bool, err error)

A found=true result may be cached.

A found=false result is returned to the caller but isn't cached, and loader errors aren't cached either.

For the same key:

goroutine A ─┐
goroutine B ─┼──→ one loader execution
goroutine C ─┘

one caller owns the shared load and executes the loader. Duplicate callers join that load instead of invoking their own loaders.

Loads for different keys remain independent.

Duplicate callers also keep their own contexts, so one of them can stop waiting after cancellation without forcing the other waiters to stop.

That solves duplicate upstream work.

It does not solve the harder problem.

Singleflight is not enough

Consider this sequence:

goroutine A: GetOrLoad("key") → loader starts

goroutine B: Set("key", newValue)

goroutine A: loader finishes

A started first.

B changed the cache later.

If A simply publishes its result when the loader returns, an older operation can overwrite newer state:

old load starts
      │
      │       Set(new value)
      │             │
      │             ▼
      │         new value stored
      │
      ▼
old load finishes
      │
      ▼
old value replaces new value

Coalescing duplicate loads does nothing to prevent this.

What is needed is ordering between load publication and cache mutations.

In pacecache, mutations such as Set, insertions through GetOrSet, Delete, and Clear act as publication barriers for the affected key or keys.

The coordination layer orders singleflight registration, mutation, and publication while tracking publication state per active key.

If a newer mutation wins before a successful loader outcome is published:

A: loader starts
      │
      │
B: Set(newValue)
      │
      └── publication barrier
                │
A: loader ends  │
      │         │
      └─────────┘
           │
           ▼
   stale outcome discarded
           │
           ▼
    ErrLoadSuperseded

the loader outcome is discarded rather than allowed to replace the newer cache state.

The loading call returns ErrLoadSuperseded.

If the loader itself returned an error, that loader error takes precedence.

Publication tracking is also per active key. Mutating another key doesn't invalidate an unrelated load merely because both keys happen to live in the same storage segment.

This distinction ended up being one of the parts of the implementation I found most interesting:

coalescing duplicate work
             ≠
ordering publication against newer state

Singleflight solves the first problem.

It doesn't automatically solve the second.

Keep observability optional

I also wanted the cache to be observable without making an observability framework part of the core API.

Stats() returns a detached snapshot of cache state and activity:

stats := cache.Stats()

fmt.Println(stats.EntryCount)
fmt.Println(stats.HitCount)
fmt.Println(stats.MissCount)
fmt.Println(stats.LoadSupersededCount)

The snapshot includes capacity and segment information together with lookup, load, cleanup, eviction, expiration, deletion, and clear activity.

There is one subtlety worth calling out.

The snapshot is assembled from independent cache segments while normal cache activity may continue. It's a detached result, but its fields aren't guaranteed to describe one globally atomic instant across the entire cache.

For applications that already use OpenTelemetry, metrics are available through the optional extra/paceotel module.

The core cache doesn't own the OpenTelemetry SDK lifecycle or exporter configuration. Those remain application concerns.

For me, that separation matters: adding a cache shouldn't force an application to adopt a particular telemetry stack.

There isn't one meaningful cache benchmark

A cache can look excellent in a throughput benchmark and still make a very different trade-off in memory consumption or hit ratio.

So I didn't want to reduce cache performance to one ns/op number.

The benchmark suite looks at three things separately.

Throughput measures concurrent read/write workloads with skewed key access and different write ratios.

Hit ratio focuses on capacity and eviction behavior under a Zipfian access pattern.

Memory measures live heap after filling caches with fixed-size keys and values at different capacities.

The repository contains the benchmark source and documents the workload configuration alongside the results.

The point isn't to produce a universal winner.

It's to make clear what is actually being measured.

Throughput, hit ratio, and memory consumption answer different questions, and an application's own access pattern ultimately matters more than a generic benchmark.

Where pacecache fits

pacecache is an in-process cache.

Every process owns its own cache state.

That makes it useful when:

• data is safe to cache locally;
• avoiding another network hop matters;
• an upstream lookup is expensive enough to benefit from cache-aside loading;
• independent cache contents across service instances are acceptable;
• the application wants a bounded local hot set.

It isn't a distributed cache.

There is no cross-process state sharing, centralized invalidation, persistence, or distributed consistency protocol.

If several service instances need one coordinated cache, Redis or another distributed system is solving a different problem.

An in-process cache and a distributed cache can both be useful, but they aren't interchangeable.

The common path should stay boring

Despite the concurrency machinery behind loading and publication, I wanted the basic API to remain unsurprising:

cache, err := pacecache.New[string, User]()
if err != nil {
    return err
}

cache.Set("user:1", user, pacecache.DefaultExpiration)

user, found := cache.Get("user:1")

A default cache has one segment, a 10,000-entry budget, and no time-based expiration.

More specialized behavior is opt-in:

• TTL and per-entry expiration;
• jitter;
• sliding expiration;
• segmentation;
• cache-aside loading;
• background cleanup;
• OpenTelemetry metrics.

The simple case shouldn't require understanding every advanced feature.

But when the advanced cases appear, their behavior should be explicit.

Feedback welcome

pacecache is open source, and the repository includes the implementation, tests, runnable examples, benchmark source, methodology, and results.

I'd especially appreciate feedback on:

• the public API;
• concurrency and publication semantics;
• cache-aside loading behavior;
• expiration behavior;
• benchmark methodology;
• real-world access patterns the current benchmarks don't represent well.

If you use in-process caches in Go services, I'd be interested to hear which trade-offs matter most in your workloads.

GitHub: https://github.com/mkbeh/pacecache