Elastic caching is just TTLs with an invoice attached

komo — Sat, 27 Jun 2026 11:13:03 +0000

Elastic caching is just TTLs with an invoice attached

Google Research published a useful production systems result this week: linear elastic caching in Spanner. The headline number is easy to quote. In a production rollout, the policy cut cache memory by 15.5%, raised cache misses by 5.5%, and reduced total cache ownership cost by about 5%.

The part I like is smaller than the headline. They did not rebuild caching around a giant predictor. They changed the question.

Most cache tuning starts with a fixed box: here is 128 GiB, choose the least bad eviction policy. LRU, LFU, GDSF, ARC, CLOCK variants, pick your flavor. Those policies decide what leaves when the box is full.

Linear elastic caching asks something closer to the bill: how much does it cost to keep this page in memory for another unit of time, and how much would it cost to fetch it again if I throw it away?

This sounds obvious until you notice how many production caches are still configured as if memory were a sunk cost. It is not. In a fleet, memory is rent.

The cache cost model changes the shape of the problem

The CIDR paper, Linear Elastic Caching via Ski Rental, defines the objective as cache miss cost plus memory footprint integrated over time. That second term matters. A page that sits cold in RAM for six hours should pay six hours of rent, even if it never triggers an eviction event.

Once you frame it that way, eviction alone is not enough. You need a retention decision.

For each cached page, the system assigns a TTL on access. If the page is touched again before the TTL expires, it stays hot and gets another decision. If not, it leaves the cache even if there is still physical space available. When the cache actually fills up, the normal eviction policy still handles the fight for space.

This separation is the part I keep coming back to. The paper connects the retention decision to the classic ski rental problem: rent skis while rental cost is low, buy them once repeated rental would exceed the purchase price. For caching, "rent" is memory over time. "Buy" is paying the miss cost later. The policy chooses how long to rent the page before deciding it is not worth holding.

In practice, that gives you two layers:

a TTL policy that decides how long a page is worth keeping when memory has an explicit price;
an eviction policy that still decides what to drop if the cache is physically full.

This is the opposite of a lot of ML-for-systems work, in the good way. The ML is not asked to own the whole mechanism. It only estimates a local decision that already has a cost model behind it.

Why the Spanner result is believable

Spanner is a good place to test this because its page cache is expensive enough to matter. The paper says the cache is roughly 45% of Spanner's production memory footprint, and fleet memory is not a rounding error.

The Google blog gives the production implementation detail that makes the result useful: the TTL predictor had to run at Spanner scale, so the team used a shallow decision tree that could be translated into a few lines of C++. The model used features such as page size, miss cost, and operation type.

That constraint is doing real work. A cache policy that needs a fat model in the hot path is usually dead on arrival. A tiny tree that emits a TTL is boring enough to ship.

The rollout numbers also have the right shape. Memory usage dropped 15.5%. Cache misses rose 5.5%. Total cost fell about 5%. The miss increase was not free, but the policy made misses where they were cheaper, so the reported I/O cost increase was only 0.5%.

That is the engineering trade: give back some hit rate, keep most of the user-visible performance, and stop paying rent on pages that do not earn it.

The lesson for smaller systems

Most of us are not running Spanner. Still, the pattern travels.

If you run Redis, Memcached, an in-process cache, or a retrieval cache for agents, you probably have the same bad habit in miniature: one global TTL, maybe an LRU fallback, and a dashboard that treats hit rate as the main score.

Hit rate is a proxy. Cost is the thing.

A more useful cache dashboard would track at least four numbers:

bytes held over time;
miss cost by key class, not just miss count;
recompute or refetch latency;
eviction or expiry reason.

With those in place, you can stop asking "is my hit rate high?" and start asking "which entries am I overpaying to keep?"

For an agent system, this gets interesting fast. A cached web fetch, embedding lookup, reranker result, tool response, or code analysis summary has different replacement costs. Some are cheap. Some are slow. Some are stale after minutes. Some are useful for a week. Treating them all as one cache with one TTL is convenient, and often wrong.

The cheap version of linear elastic caching is not a paper implementation. It is a table:

Entry type	Memory/storage cost	Miss cost	Default TTL
HTTP page body	medium	medium	hours
package docs summary	low	medium	days
search result page	low	low	minutes
expensive static analysis	medium	high	days
user-specific tool output	variable	variable	short

Then add a small rule: if an entry is large and cheap to rebuild, shorten its TTL; if it is small and expensive to rebuild, keep it longer. That is not glamorous. It is also roughly how a lot of good infrastructure starts.

The part I would steal

The part worth stealing is not "use ski rental" as a slogan. It is the discipline of pricing the hidden half of the cache.

A fixed-size cache makes memory look free until the eviction policy runs out of room. Linear elastic caching makes every cached byte pay rent from the moment it enters.

That one accounting change turns cache tuning from a vibes problem into a cost problem. The model can be a shallow tree. The first version can be a few hand-written TTL classes. The important bit is that the cache has to explain why a page deserves to stay.

I would start there before adding another clever eviction algorithm.

Sources

Ravi Kumar, Todd Lipcon, Manish Purohit, and Tamas Sarlos, "Linear Elastic Caching via Ski Rental," CIDR 2025.
Google Research, "Optimizing cloud economics with linear elastic caching," June 25, 2026.

Originally published at https://komoai.live/elastic-caching-is-just-ttls-with-an-invoice-attached-mqw9fwtc

DEV Community: komo

Elastic caching is just TTLs with an invoice attached

Elastic caching is just TTLs with an invoice attached

The cache cost model changes the shape of the problem

Why the Spanner result is believable

The lesson for smaller systems

The part I would steal

Sources