The real economics of Redis tiered storage, and why you might be paying too much for hot data
The concept of splitting data resident in a Redis™ cluster between RAM and slower/cheaper non-volatile storage is something that's firmly gaining traction: all the major Redis-like vendors either offer some way of letting data spill out of memory directly in their product, or as a managed service. The simultaneous explosion in LLM-related caching infrastructure and the price of DRAM is heightening interest in hybrid memory caches. Let's examine how the two most prominent vendors in this space: AWS/ElastiCache and Redis themselves, address tiered storage, and ask if the price/performance trade is worthwhile?
The economics of Redis tiered storage
Redis and most of its clones spent the first decade or so of their existence as pure in-memory data stores; sure, it gained some persistence capabilities but these were, and continue to be things which facilitate HA/DR rather than something that sits in the hot path to handle client requests.
In a nutshell, tiered storage lets Redis host datasets which may be split between memory, disks or other persistent storage in a way that's completely transparent to the client; moving data between layers in the storage hierarchy is handled internally. This allows operators to trade off some performance for cheaper storage. The cost of memory and proliferation of Redis datasets which can't practically fit into memory anyway, have been a massive catalyst for the adoption of storage hierarchies in production Redis deployments.
ElastiCache Data Tiering
AWS introduced Data Tiering for ElastiCache for Redis in 2021, using r6gd instances with local NVMe SSD storage alongside RAM. The basic idea is straightforward: keep the hottest data in memory, while less frequently accessed data is moved onto local SSD. From the application's perspective, it's still Redis; the movement between tiers is handled by the service. ElastiCache evicts values to SSD on an LRU basis, and importantly values are always held in memory. Amazon also documents some important limitations, including:
Items > 128MiB are not eligible for SSD storage
You can't back up a tiered ElastiCache setup to S3
This is a useful model if you have a dataset substantially larger than the amount of RAM you'd otherwise be willing to pay for, particularly when only a relatively small proportion of the data is being accessed regularly.
Conceptually, the hierarchy looks like this:
┌──────────────┐
│ Client │
└──────┬───────┘
│
▼
┌──────────────┐
│ RAM │ ← performance tier
└──────┬───────┘
│
▼
┌──────────────┐
│ SSD │ ← capacity tier
└──────────────┘
Adding an SSD tier can make a large dataset dramatically cheaper than putting the entire dataset in DRAM, but it doesn't change the fundamental economics of the hot portion. If your workload requires hundreds of gigabytes of data to remain in RAM, you still need to pay for it.
How much does ElastiCache data tiering cost then?
Suppose you have a 1 TB dataset, but only 10% of that data is being accessed frequently enough that you need it at memory-like latency (AWS' own docs allude to a max hot percentage of 10-20% as the sweet spot for ElastiCache data tiering). The smallest single node that'll fit that dataset is a cache.r6gd.12xlarge, which in us-east-1 sets you back $6,831.34 a month, before any savings plans are applied. You don't get to directly pick the hot percentage with ElastiCache so with ~318MiB of RAM that actually gives you roughly a 26% RAM:SSD ratio. In a real setup, single-node throughput will become an issue at some point, so you may pick more, smaller instances, but the cost/GB for cache.r6gd instances and the RAM:SSD ratio is pretty constant across the fleet. You'll want HA too, which adds a multiplier to that figure from earlier (remember also that SSDs aren't replicated across AZs or anything like that, and that you can't back up these clusters' data to S3).
Redis Flex Tiered Storage
Redis has been working on this problem for considerably longer, originally through Redis on Flash and subsequently through Auto Tiering. The current incarnation is Redis Flex, which continues the same basic model: use RAM for the hot portion of the dataset and SSD for the colder portion.
Both products are answering essentially the same question:
How can we make a Redis dataset larger than RAM without requiring the entire dataset to be backed by RAM?
For some workloads the answer produces very attractive economics, but it's constrained by Redis' original underpinnings as an in-memory db. A more on-point question is:
How much RAM does the workload actually need in the first place?
So, how much does “hot” actually cost?
Let's stick with our 1TB dataset. Unsurprisingly, a tiered Redis deployment can be considerably cheaper than a 1 TB all-memory setup.
Deploying on AWS in us-east-1, at the time of writing a fully in-memory 1TB Redis cluster will cost you $17,082/month, whereas 1TB Flex Pro with 10% hot in the same region goes for $3,306. A ~5x saving.
If you crank up the hot percentage to 50% (a documented product limit, BTW), and the economics change substantially; you're getting much closer to simply paying for a very large memory-optimized Redis deployment, while still carrying the complexity of a second storage tier. That takes the monthly total to $11,870.
That's why the percentage of your dataset which needs to be "hot" is arguably more important than the headline capacity of the tiered instance.
How hot do you actually need it?
Both ElastiCache Data Tiering and Redis Flex are grounded in Redis' own heritage, which places RAM as the primary data store and everything else as a supplementary extension. Historically, this has worked well for many workloads which need to reside all or mostly in memory and need tight upper bounds on latency.
That being said, there's a much more common case where the requirement isn't, "I need my entire database to behave as though it were in memory", but "I need my working set to be fast", however you care to define fast.
In-memory vs on-machine
Modern NVMe storage is extremely fast, particularly when it is directly attached to the machine doing the processing. It's not RAM-fast and for workloads needing that tight upper-bound on latency, you'll still have to pay through the node for your hardware, but for many use cases, a modest amount of RAM combined with a large local NVMe device is a perfectly acceptable storage hierarchy.
A different storage hierarchy
Invar was built from the ground up as a persistent store, based on LSM trees; it uses SlateDB in production, and fast storage is layered on top as necessary. Local NVMe volumes provide a large, fast working-storage layer, while object storage provides the durable backing store. RAM's primary function here is for page-caching SSTables which the Invar process reads through Foyer. While you're free to choose whatever hardware makes sense for your workload, the guiding philosophy is that aside from the network hop to S3, the
The resulting hierarchy looks markedly different:
┌──────────────┐
│ Client │
└──────┬───────┘
│
▼
┌──────────────┐
│ RAM │ ← page cache
└──────┬───────┘
│
▼
┌──────────────┐
│ NVMe │ ← local working storage
└──────┬───────┘
│
▼
┌──────────────┐
│ S3 │ ← source of truth
└──────────────┘
The distinction is important: RAM and NVMe aren't two tiers of the database which have to contain the complete working dataset. They're best-effort performance and locality layers in front of the durable store.
If something isn't in RAM, it can be read from local NVMe. If it isn't available locally, SlateDB can retrieve it from the backing object store.
That means the amount of RAM you provision is primarily a function of your workload's working set, rather than the total size of your database.
Yes, a network hop from an instance to S3 can potentially add a nontrivial amount of latency but modern S3 offerings (think Tigris and S3 Express) are chipping away at this, and provisioning large NVMe volumes to mitigate cache misses is orders of magnitude cheaper than provisioning RAM.
What does this mean for cost?
Let's revisit our 1TB, 10% hot case. Our suggestion is to go for storage-optimised instances such as the i8g / i8ge family. We could over-provision a i8ge.12xlarge which has 30x the NVMe capacity of the entire dataset and approximately the same ~300GiB of RAM that the r6gd.cache.12xlarge has, which as a straight on-demand instance $4,157.50 a month in us-east-1. On top of that, Invar persists all data to S3; 1TB worth of regular S3 costs $23.55 a month (since the entire dataset fits in NVMe, aside from background coordination, the S3 API call volume will be negligible for a dataset that will statistically never fall out of the node, unless it dies).
That being said... this is where the comparison becomes less about "which tiered Redis product is cheapest?" and more about the underlying hardware economics.
Memory is expensive. Local NVMe is dramatically cheaper per gigabyte, and object storage is orders of magnitude cheaper than both. Can we do better than 4 grand a month? If you don't need pure at-memory speed for everything, we can still fit our entire dataset in NVMe on a i8ge.xlarge with around a TB of room to spare for $173.81 a month on-demand (plus the ~25 bucks for S3). Most people would agree that 7 grand a month for ElastiCache vs ~$200 is a huge difference!
We're not claiming that NVMe can somehow replace RAM; it can't. A workload which genuinely requires memory-speed access to a huge proportion of its dataset will still benefit from having a huge proportion of that dataset in memory.
The question is whether your workload actually has that characteristic, or whether you're just accepting a RAM + SSD storage hierarchy purely because Redis and its descendants have historically only supported storing data in memory and block devices?
If you have a 1 TB database with a 50 GB working set, buying a platform which expects you to provision hundreds of gigabytes of RAM and then uses SSD to hold the rest may be solving a different problem from the one you actually have.
With an on-machine architecture, you can instead provision enough RAM to keep the working set comfortably cached and use inexpensive local NVMe for the rest.
S3 is cheaper again, and in Invar it provides the ultimate durable backing store rather than being something that needs to sit in the request path for every operation. The central point is simply that the capacity hierarchy extends beyond the machine itself- S3's durability is another quiet advantage of Invar, but I'll save that for another post :)
The point isn't that everything should be on NVMe
To reiterate, none of this implies that established tiered Redis solutions are inherently a bad idea. If your application genuinely has a large hot set and needs Redis's characteristic memory-speed access patterns, keeping that hot set in RAM and putting the rest on SSD can make a lot of sense.
The more interesting question is whether your database needs to be an in-memory store at all. Invar has designed around that distinction as a data store which speaks the Redis protocol, but isn't constrained by 20 years of architectural decisions that Redis itself has to carry.
Instead of starting with a large RAM-backed Redis instance and asking how much data you can push out onto cheaper storage, Invar starts with object storage as the source of truth, and local storage as the primary working medium, using RAM and the OS page cache to accelerate the part of the dataset your workload actually needs.
Try Invar
Invar is open source and Redis-compatible, so the easiest way to find out whether this model is a good fit for your workload is to try it out.
We're also preparing the first Invar Enterprise release: a production-oriented Helm deployment with a validated HA configuration, externally managed compaction/GC, extended ops & observability tooling and commercial support options.
Top comments (0)