Shahid Shaikh

Posted on Feb 4

Redistill - High Performance Redis Compatible Key Value Database

#redis #rust #database #programming

Redis has always been one of my go-to KV databases. I strongly believe in — and advocate for — a memory-first, disk-later approach when designing high-throughput, low-latency systems. Redis is fast and durable, but I kept wondering: can we build something Redis-like that’s even faster?

That curiosity pushed me to experiment deeply with Redis and the RESP protocol. Eventually, I decided to “distil” Redis — strip it down to the essentials and build a lighter version that supports only the commands I use most often (Strings and Hashes) — and see if we could push throughput and latency even further.

Turns out, we did.

I’m happy to announce that I’m working on an open-source project called Redistill — a distilled Redis. It’s written in Rust and based on benchmarks run on AWS c7i.16xlarge (Intel, 64 cores, 128GB RAM) using memtier_benchmark, I can confidently say this is the fastest single-instance, Redis-compatible database available today.

Project link: https://github.com/redistill-io/redistill

This is just the beginning. The project is barely 8 weeks old. I still need to figure out clustering and other production-ready features, but so far, as a single instance, this is ready to use ina production environment.

Motivation

The entire motivation behind the project was to build my own KV database that works faster than Redis and supports the commands I use frequently. It doesn't need to be versatile like Redis, but it does one thing best - the fastest KV database possible.

I also wanted to learn Rust, and this was a good opportunity to revisit the fundamentals of system design and build a database from scratch (not entirely scratch, I use RESP protocol).

Key Characteristics

Redistill supports the RESP protocol, so there is no need for code changes on the client side. Your existing code, which works with Redis, will work with Redistill, assuming you use commands Redistill supports.

Below are some key features:

High-performance in-memory key-value database
Redis protocol compatible (RESP) - works with all Redis clients
9.07M operations/second - 4.5x faster than Redis, 1.7x faster than Dragonfly
5x lower latency (p50: 0.48ms vs Redis 2.38ms)
Multi-threaded architecture with lock-free reads
Optional persistence - Snapshot support for warm restarts (disabled by default for max performance)
Production-ready security and monitoring features

Why not all Redis commands?

This was intentional.

I wanted a KV database for my personal and work uses. I used String and Hash commands almost 90% of the time. So keeping it lightweight was the design and not a restriction.

Will I add more commands in the future? Of course, this is a FOSS project, so I'll let contributors add the commands as they seem fit.

Why Redistill?

High-Performance KV Database - Fastest in-memory key-value database available
Maximum Single-Instance Performance - 4.5x faster than Redis, 1.7x faster than Dragonfly per instance
Lower Latency - Sub-millisecond p50 latency (0.48ms)
Cost Efficient - 50-83% infrastructure savings (fewer instances needed)
Drop-in Compatible - Works with existing Redis clients
Production Ready - TLS, authentication, monitoring, health checks
Multi-threaded - Utilises all CPU cores efficiently
Optional Persistence - Snapshot support for warm restarts when needed (disabled by default)

Quick Start

We recommend using Docker to quickly spin up Redistill in your machine or server.

# Run with default settings
docker run -d --name redistill -p 6379:6379 shahidontech/redistill:latest

# Test it works
redis-cli ping
# PONG

redis-cli set hello world
# OK

redis-cli get hello
# "world"

We also have other methods, such as Homebrew and building from source.

Performance

Competitive Benchmark (c7i.16xlarge)
Independent comparison on AWS c7i.16xlarge (Intel, 64 cores, 128GB RAM) using memtier_benchmark with production-like configuration:

Test Configuration:

Duration: 60 seconds
Threads: 8, Connections: 160 (20 per thread)
Pipeline: 30, Data size: 256 bytes
Workload: 1:1 SET: GET ratio

Metric	Redistill	Dragonfly	Redis	vs Redis	vs Dragonfly
Throughput	9.07M ops/s	5.43M ops/s	2.03M ops/s	4.5x	1.7x
Bandwidth	1.58 GB/s	923 MB/s	337 MB/s	4.7x	1.7x
Avg Latency	0.524 ms	0.877 ms	2.000 ms	3.8x faster	1.7x faster
p50 Latency	0.479 ms	0.807 ms	2.383 ms	5.0x faster	1.7x faster
p99 Latency	1.215 ms	1.975 ms	2.959 ms	2.4x faster	1.2x faster
p99.9 Latency	1.591 ms	2.559 ms	4.159 ms	2.6x faster	1.6x faster

Key Observations:

Redistill processed 544M total operations (2.7x more than Dragonfly, 4.5x more than Redis)
Consistent low latency across all percentiles
No errors or connection issues across all systems

Use cases

Redistill is a high-performance key-value database perfect for applications requiring maximum speed and minimal latency.

Perfect For
Session Storage

1M+ operations/second
Sub-millisecond latency
Automatic TTL expiration
60% cost reduction vs alternatives

API Response Caching

95%+ cache hit rates
50-150x faster than database queries
Automatic memory management
LRU eviction is built in

Rate Limiting

Millions of counters
TTL-based cleanup
High write throughput
Perfect for API gateways

Real-time Leaderboards

Fast reads for rankings
Periodic score updates
Can rebuild from the database
Sub-millisecond queries

Not Recommended For

Critical persistent data (optional snapshots available, but not real-time durability)
Financial or transactional data (use ACID-compliant database)
Data that cannot be regenerated (use a database)

Examples

The Redistill uses the RESP protocol to communicate, so any Redis client library should be able to connect to Redistill and communicate as they are communicating with Redis with the same commands.

For example, in Python:

import redis
r = redis.Redis(host='localhost', port=6379, password='your-password')
r.set('key', 'value')
value = r.get('key')

In Node.js:

const Redis = require('ioredis');
const redis = new Redis({host: 'localhost', port: 6379, password: 'your-password'});
await redis.set('key', 'value');

DEV Community