DEV Community: HotKV Ltd

Redis benchmark: HotKV vs Valkey and Dragonfly in a three-node cluster

HotKV Ltd — Thu, 05 Nov 2026 19:00:00 +0000

This Redis benchmark runs HotKV, Redis, Valkey and Dragonfly as three-node clusters on AWS. Our earlier benchmarks ran each server as one instance on a whole machine. That layout suits HotKV, which uses every core in one process. Redis and Valkey run commands on one thread per process, and they are commonly scaled across cores with many processes, usually as a cluster. So we measured every product the way a cluster deploys it.

In short. In this Redis benchmark on three AWS servers, HotKV had the highest throughput on 8 of 9 commands: 92.3 million GETs per second against 57.3 million for Redis, 52.9 million for Valkey and 22.5 million for Dragonfly. Redis and Valkey were ahead on APPEND. The load generators were near their limit on some commands, so the figures are lower bounds.

Why we ran this Redis benchmark as a cluster

A single-instance benchmark, like the one in our launch article, asks how much one server process can do with a whole machine. That is a reasonable question for HotKV and Dragonfly, which use all cores in one process. It says less about Redis and Valkey. Their commands run on one thread per process (I/O threads can offload network work), so a 48-vCPU server is normally used by running dozens of processes.

We do not want our numbers to depend on a layout that suits one architecture better than another. A cluster is common ground. All four products can run as one, and for Redis and Valkey it is a usual way to use many cores. We ran this benchmark to see how the comparison looks when each product is deployed that way.

What we tested

Every product ran as a cluster on the same three servers, one product at a time, driven by the same two load generators. The sections below give the machines, each layout and the procedure.

Machines

Servers. Three c8in.12xlarge instances (Intel Xeon 6975P-C, 48 vCPUs, 75 Gbit/s network) in one AWS cluster placement group in eu-south-2a, running Ubuntu 24.04. Every server process was pinned to CPUs 0 to 39.
Load generators. One c8in.48xlarge running 16 memtier_benchmark processes of 12 threads each, and one c8in.12xlarge running 4 processes of 12 threads. They were started together and the operations per second of all processes were added up.
Tuning. All five machines got the same host tuning: the repository's tune_server.sh, receive packet steering over all CPUs, the network card's receive ring at its maximum, a larger softirq budget and a listen backlog of 65,535.

Servers and layout

HotKV. One process per server with 40 threads, in cluster mode (3 primaries). The build is 0.3.0 plus a change that adds the pipeline-prefetch setting, and it ran with pipeline-prefetch yes. The setting is off by default. It looks ahead in a client's pipeline and asks the CPU for the memory the next commands will read.
Redis 8.10.3 and Valkey 9.1.3. 40 single-thread processes per server, so 120 primaries, in cluster mode. Each process had a core to itself. Their own batch prefetching was left at its default.
Dragonfly 2.0.2. One process per server with 40 threads, in cluster mode, built from source.
Persistence. Off on every server.

Workload and procedure

The data set was 5,000,000 keys with 256-byte values, and pipeline depth was 100. memtier_benchmark ran in cluster mode, so each client talks to the right node. Each command ran for 15 seconds, twice, and we report the median of the two. The data set was loaded again, identically, before every run, so a command never ran on top of the previous command's changes. The scripts and every run are in the benchmark repository.

Results

Millions of operations per second, median of two runs. The highest value in each row is in bold.

Command	HotKV	Redis 8.10.3	Valkey 9.1.3	Dragonfly 2.0.2	HotKV vs Redis	HotKV vs Valkey
GET	92.3	57.3	52.9	22.5	1.61x	1.74x
SET	76.1	55.0	48.5	25.2	1.38x	1.57x
SETNX	86.1	60.7	57.0	27.5	1.42x	1.51x
INCR	119.2	118.4	104.6	30.8	1.01x	1.14x
SADD	51.7	40.4	33.8	22.7	1.28x	1.53x
RPUSH	47.1	29.3	29.0	18.7	1.61x	1.62x
HSET	38.4	28.8	27.2	18.4	1.33x	1.41x
LPUSH	31.1	23.7	24.0	16.9	1.31x	1.30x
APPEND	35.0	41.7	40.9	10.4	0.84x	0.86x

Against Dragonfly, HotKV was ahead on all nine commands, from 1.85 times (LPUSH) to 4.09 times (GET).

How to read the ratios

A throughput figure from a load test is a floor when the load generators are the limit. If a load generator is at 100 percent CPU, the server may have been able to take more. The figure then shows only that it took at least this much. A ratio between two floors is not exact, because either side could be higher.

We flag every figure where the main load generator ran above 85 percent CPU. These were GET, SETNX and INCR for HotKV, and GET, SET, SETNX, INCR, SADD, HSET and APPEND for Redis and Valkey. On those figures, read the ratio as a rough size and not a precise one.

On SADD and HSET, HotKV's figure is not flagged but Redis's and Valkey's are. Their true throughput may be higher, and HotKV's lead smaller than shown. The same applies to APPEND, where it would make their lead over HotKV larger than shown.

RPUSH and LPUSH have no flag for any server, so they give the cleanest comparison. HotKV was 1.3 to 1.6 times ahead of Redis and Valkey on them.

What the results show

Three things stand out in the table, and the first is about Redis and Valkey rather than HotKV.

Redis and Valkey scale well as clusters

With 120 single-thread processes, Redis served 57 million GETs per second and 118 million INCRs per second across three servers. That confirms the cluster layout suits them. It is also why HotKV's margin here is smaller than in our single-instance runs.

Those runs compared a design that uses all cores in one process with products that were not run in the layout they are commonly scaled in. This Redis benchmark gives Redis and Valkey that layout. HotKV's lead over them narrows to between 1.0 and 1.7 times on eight commands.

HotKV had the highest throughput on 8 of 9 commands

The lead was widest on GET and RPUSH against Redis (1.6 times) and on GET against Valkey (1.7 times). On RPUSH and LPUSH the load generators had room for every server, so those figures are limited by the servers. On INCR, HotKV and Redis were level within 1 percent, and the load generators, not the servers, set the limit for all three.

Redis and Valkey were ahead on APPEND

HotKV served 35.0 million APPENDs per second against 41.7 million for Redis and 40.9 million for Valkey. We have not found the reason. APPEND grows the dataset by about 10 GB per second across the cluster, so the run is also a test of memory allocation. A 30-second APPEND run ran the servers out of memory.

We profiled HotKV on APPEND and saw page faults, TLB shootdowns and lock contention. We have not shown which of them accounts for the gap.

Limitations

These are HotKV Ltd's measurements of the versions and configuration above. They are not a claim about any other version, configuration or workload of these products.

The test setup

Load generators near their limit. Many figures are lower bounds, as the section on reading the ratios explains. HotKV's APPEND figure is not one of them, because the load generators were at about 30 percent CPU with it.
Layouts we chose. We ran Redis and Valkey as one single-thread process per core and did not tune either product beyond that. Other layouts or tuning could change their figures. If you think a setting here disadvantages a product, tell us and we will check it and publish the result.
Two runs, one region. Each figure is the median of two 15-second runs on one set of machines in one region. We did not measure variation between machines or days.

What was not measured

Latency. We measured throughput only here. A separate Redis latency benchmark measures latency.
Persistence, replication and failover. All servers ran with persistence off and no replicas.
Other setups. Three servers and one workload shape. A larger cluster, other value sizes or other pipeline depths can change the ratios.

Redis, Valkey and Dragonfly are trademarks of their respective owners. They are named only to identify the software we measured, and this implies no sponsorship, endorsement or affiliation.

Reproduce it

The sweep script, the exact command sequence, the host tuning and every raw result are in the benchmark repository. The sweep script is harness/cluster_bench_sweep.sh. A chart of the same figures is on the performance page.

Try it yourself

HotKV Cloud runs HotKV clusters for you. Start on HotKV Cloud, or talk to sales to benchmark a cluster against your own workload.

Frequently asked questions

Why run a Redis benchmark as a cluster?

Redis and Valkey run commands on one thread per process, so a single process cannot use a large machine. They are commonly scaled across cores by running many processes as a cluster. A cluster benchmark measures each product in a layout like that, so the comparison does not rest on a single-instance deployment alone.

Did HotKV have the highest throughput in cluster mode?

On 8 of 9 commands, yes. HotKV led Redis by 1.0 to 1.6 times and Valkey by 1.1 to 1.7 times, and Dragonfly by 1.85 to 4.09 times. Redis and Valkey were ahead on APPEND. On INCR, HotKV and Redis were level within 1 percent, and the load generators set the limit.

Why were Redis and Valkey ahead on APPEND?

We do not know yet. APPEND grows the dataset by about 10 GB per second across the cluster, so the run also tests memory allocation. A profile of HotKV showed page faults, TLB shootdowns and lock contention, but we have not shown which of them accounts for the gap.

What is the pipeline-prefetch setting?

It makes HotKV look ahead in a client's pipeline and ask the CPU for the memory the next commands will read, so those reads overlap. It changes no reply. It is off by default, and HotKV ran with it on in this benchmark.

Can I reproduce this Redis benchmark?

Yes. The sweep script, the exact command sequence, the host tuning and every raw result are in the benchmark repository, and the same figures are charted on the performance page. You need three servers, one or more load generators and the four products installed by the harness.

Does this replace the single-instance results?

No. The two answer different questions. A single-instance run shows what one server process does with a whole machine. A cluster run shows what three machines do when each product is laid out as a cluster. Both are on the performance page.

Redis latency benchmark: HotKV vs Valkey and Dragonfly

HotKV Ltd — Thu, 05 Nov 2026 07:00:00 +0000

This Redis latency benchmark measures how long HotKV, Redis, Valkey and Dragonfly take to answer at a fixed request rate. Our earlier benchmarks measured throughput, and a reader rightly pointed out that throughput is not everything. So we held the load constant, raised it step by step, and recorded the median (p50) and the 99th percentile (p99) at each step. We also measured pub/sub delivery time.

In short. In this Redis latency benchmark on AWS, HotKV and Dragonfly both held 1.6 million GETs per second, Redis with 8 I/O threads held about 0.8 million, Valkey with 8 I/O threads about 0.4 million, and single-thread Redis and Valkey about 0.2 million. From 200,000 to 1.6 million requests per second HotKV's p99 stayed between 0.59 and 0.76 ms, against 0.71 to 1.48 ms for Dragonfly. At light load the multi-threaded products tied on p99, and pub/sub delivery was a tie.

Why we ran a Redis latency benchmark at a fixed rate

A throughput test pushes a server as hard as it will go and reports how much it served. That says little about how long one request waits. A latency test fixes how many requests per second the clients send, and measures the time each takes. The server is then at the same load as every other product, so the times can be compared.

This kind of test has a catch. memtier_benchmark is a closed-loop tool, so a client waits for a reply before it sends the next request. If a server cannot keep up, the client simply sends fewer requests, and the latency of the ones it did send looks better than it should. We therefore count a step as sustained only when the achieved rate is at least 97 percent of the rate we asked for. Steps that fail the test are marked "not sustained" in the tables and their latency is not shown.

What we tested

Machines

Servers and load generator. c8a.12xlarge instances (AMD EPYC 9R45, 48 cores, no SMT, 96 GiB, 22.5 Gbit/s network) in one AWS cluster placement group in eu-south-2a, running Ubuntu 24.04. One instance ran the load generator and one ran the server.
One product at a time. Each product ran alone on the server, so nothing overlapped and nothing shared the network. Every server process was pinned to CPUs 0 to 39, with persistence off and protected mode off.

Products

HotKV 0.3.0 with 40 threads.
Dragonfly 2.0.2 with 40 proactor threads.
Redis 8.10.3 and Valkey 9.1.3, each run twice, once in the default single-thread setup and once with --io-threads 8.

The launch line and the live CONFIG GET output of every run are saved with its results.

Workload and procedure

The data set was 1,000,000 keys with 256-byte values, and pipeline depth was 1, so each request waits for its own reply. The load was GET only, and a mix of 1 write to 10 reads. Each step ran for 20 seconds, twice, and we report the mean of the two runs. memtier_benchmark used a per-connection rate limit and recorded an HDR histogram. We ran two connection counts.

816 connections (24 threads of 34 connections), for rates from 20,000 to 2.4 million requests per second.
48 connections (4 threads of 12 connections), for rates up to 300,000 requests per second. Above that, 48 connections cannot reach the rate, because each connection can only send as fast as it gets replies.

Results

The GET tables come first, one for each connection count, followed by pub/sub.

At 816 connections

Each cell is p50 / p99 in milliseconds, for GET. "NS" means not sustained, and the rate it did reach is in brackets. A dash means the sweep for that product had already stopped.

Requests/s	HotKV	Dragonfly	Redis 1 thread	Redis 8 I/O threads	Valkey 1 thread	Valkey 8 I/O threads
19,584	0.38 / 1.81	0.48 / 1.90	1.63 / 5.29	0.68 / 2.07	1.59 / 5.13	0.76 / 2.63
99,552	0.24 / 0.76	0.27 / 1.76	2.60 / 5.12	0.40 / 0.98	2.65 / 5.13	1.71 / 3.19
199,920	0.25 / 0.59	0.25 / 1.34	2.91 / 5.60	0.55 / 1.11	3.02 / 5.81	1.11 / 2.15
399,840	0.27 / 0.63	0.27 / 1.48	NS (258k)	0.62 / 1.23	NS (259k)	1.24 / 1.49
799,680	0.28 / 0.75	0.28 / 1.26	-	0.68 / 1.18	-	NS (670k)
1,199,520	0.28 / 0.67	0.28 / 1.05	-	NS (1,078k)	-	-
1,599,360	0.28 / 0.64	0.28 / 0.71	-	-	-	-
2,399,856	NS (2,284k)	NS (2,281k)	-	-	-	-

The mixed workload gave the same picture. HotKV's p99 was 0.54 to 0.75 ms from 200,000 to 1.6 million requests per second, and the points each product could sustain were the same as for GET. All of it is in the results folder.

How to read the single-thread rows

The 1.6 to 3 ms medians of single-thread Redis and Valkey at 816 connections are not their steady-state latency. With a rate limit, memtier_benchmark makes all 816 connections fire at the same moment. A single-thread server has to work through that burst of 816 requests one after another, so the median of a request includes its place in the queue. We ran the 48-connection test below to remove this effect.

At 48 connections

The cells have the same meaning as above, for GET only.

Requests/s	HotKV	Dragonfly	Redis 1 thread	Redis 8 I/O threads	Valkey 1 thread	Valkey 8 I/O threads
9,984	0.14 / 0.54	0.14 / 0.37	0.21 / 0.89	0.17 / 0.92	0.21 / 0.91	0.20 / 0.81
49,968	0.10 / 0.28	0.11 / 0.28	0.21 / 0.41	0.13 / 0.28	0.21 / 0.43	0.21 / 0.44
99,984	0.09 / 0.28	0.10 / 0.28	0.20 / 0.40	0.13 / 0.28	0.20 / 0.38	0.19 / 0.35
199,968	0.08 / 0.27	0.08 / 0.27	0.20 / 0.35	0.10 / 0.27	0.21 / 0.36	0.11 / 0.28
300,000	0.08 / 0.27	0.08 / 0.27	NS (231k)	0.09 / 0.26	NS (233k)	0.09 / 0.26

Pub/sub delivery time

We measured the time from a publish to its delivery to a subscriber, with 64 publishers, 100 subscribers and a 128-byte payload, using pubsub-sub-bench. Each message went to one subscriber. Cells are p50 / p99 in milliseconds.

Deliveries/s	HotKV	Dragonfly	Redis 1 thread	Redis 8 I/O threads	Valkey 1 thread	Valkey 8 I/O threads
about 21,000	0.10 / 0.71	0.10 / 0.65	0.11 / 0.74	0.11 / 0.62	0.11 / 0.76	0.12 / 0.78
about 107,000	0.10 / 1.02	0.12 / 0.92	0.10 / 0.80	0.11 / 0.63	0.10 / 0.82	0.10 / 0.86
about 215,000	0.11 / 1.20	0.13 / 1.13	0.51 / 1.03	0.20 / 0.73	0.57 / 1.07	0.12 / 1.22

All products delivered in the same range. At the highest load the single-thread products had a higher median, 0.51 and 0.57 ms against 0.11 ms for HotKV.

We also had to patch pubsub-sub-bench. It compared a publish time in milliseconds with a receive time in microseconds. The patch is in the results folder.

What the results show

The tables show a lead for HotKV and Dragonfly at high load, a tie at light load, and a ceiling for single-thread Redis and Valkey.

HotKV and Dragonfly kept a low tail to 1.6 million requests per second

Both sustained 1.6 million GETs per second at 816 connections, with a median of 0.28 ms. HotKV's p99 was between 0.59 and 0.76 ms from 200,000 requests per second up. Dragonfly's was between 0.71 and 1.48 ms, and the two were close at 1.6 million (0.64 against 0.71 ms). Redis and Valkey with I/O threads stopped sustaining the rate at lower loads, and single-thread Redis and Valkey stopped at about 260,000.

At light load the multi-threaded products tie

At 48 connections, up to 300,000 requests per second, HotKV, Dragonfly and Redis with 8 I/O threads all had a p99 of about 0.27 ms. HotKV had the lowest median, 0.08 to 0.09 ms from 100,000 requests per second, with Dragonfly at 0.08 to 0.10 ms. A difference of 0.01 ms is within what other parts of the setup can cause, so we do not claim it as a win.

Single-thread Redis and Valkey ran out at about 230,000 a second

At 48 connections they had a median of about 0.2 ms and a p99 of 0.35 to 0.43 ms, and they did not sustain 300,000 requests per second (about 231,000 and 233,000). That is the ceiling of one thread on this hardware and workload. It is why Redis and Valkey are usually run as many processes or as a cluster, as in our cluster benchmark.

Limitations

These are HotKV Ltd's measurements of the versions and configuration above. They are not a claim about any other version, configuration or workload of these products.

The test setup

One server, one load generator. Two repeats per step, on one set of machines on one date. We did not measure variation between machines or days.
Layouts we chose. Redis and Valkey ran in their default single-thread setup and with --io-threads 8, and we did not tune either beyond that. Other settings could change their figures. If you think a setting here disadvantages a product, tell us and we will check it and publish the result.
The tool's burst effect. At 816 connections, the medians of single-thread products are inflated by the way memtier_benchmark rate limiting works, as explained above.

What was not measured

Data. Larger or skewed values, memory per key, and persistence. Persistence was off for every server.
Modes. Cluster mode, sharded pub/sub, TLS, replication and failover.
Pub/sub fan-out. One message to many subscribers.

Redis, Valkey and Dragonfly are trademarks of their respective owners. They are named only to identify the software we measured, and this implies no sponsorship, endorsement or affiliation.

Reproduce it

The sweep scripts, the launch line and live configuration of every server, the pubsub-sub-bench patch and every raw result are in the benchmark repository. The latency script is harness/lat_sweep.sh, and the pub/sub script is harness/ps_sweep.sh. The raw folder holds one archive per test with the memtier_benchmark JSON, the HDR histograms and the CPU use of the server for every run.

Try it yourself

HotKV Cloud runs HotKV for you. Start on HotKV Cloud, or talk to sales to benchmark HotKV against your own workload.

Frequently asked questions

Is HotKV faster than Redis in latency?

Under load, at the rates we tested, yes. At 816 connections and 200,000 requests per second HotKV had a p99 of 0.59 ms, Redis with 8 I/O threads 1.11 ms and single-thread Redis 5.60 ms. At light load with 48 connections the multi-threaded products tied on p99, at about 0.27 ms.

Why did you hold the request rate fixed?

At a fixed rate every product carries the same load, so the latencies can be compared. A throughput test shows how much a server serves at its limit, which says little about how long one request waits.

What does "not sustained" mean?

The product achieved less than 97 percent of the request rate we asked for. memtier_benchmark is a closed-loop tool, so an overloaded server makes the clients send less and the latency of the remaining requests looks better than it is. We show no latency for those steps.

How did pub/sub delivery time compare?

At the three loads we tested, all products delivered in the same range, with a median of about 0.1 ms at the two lower loads. The table above has every figure.

Can I reproduce this Redis latency benchmark?

Yes. The scripts, the configurations and every raw result are in the benchmark repository. You need two servers on the same network, a load generator, and the four products.

Vector search: HotKV vs Redis 8 vector sets, from 1 to 100 clients

HotKV Ltd — Thu, 05 Nov 2026 01:39:46 +0000

Vector search sits underneath many AI features: retrieval for RAG, semantic caching, agent memory. Redis 8 ships vector sets, a beta data type, built into the server, so it is the obvious comparison. We loaded the same OpenAI embeddings into HotKV and into Redis 8.10.2, ran the same 1,000 queries against each with 1 to 100 concurrent client connections, and measured how accurate the answers were, how many queries per second each engine served, and how long each query took.

In short. On 999,000 vectors, at a minimum recall of 95%, HotKV served 1.4 times the queries per second of Redis with one client and 7.9 times with 100 clients (51,146 against 6,510). At 99% recall the ratios were 1.8 and 5.4. HotKV's median latency was lower at every client count. Redis's throughput levelled off by 32 clients, while HotKV's kept growing to 100. HotKV loaded the vectors in 188 s against 935 s for Redis, which was pinned to four vCPUs for its load, and its recall was higher at every search depth. This is one dataset on one machine, the HotKV build is an unreleased development build, and we did not measure Redis's Query Engine or its 8-bit mode, so read it as a measurement, not a ranking.

What we tested

Data

The first 999,000 rows of dbpedia-entities-openai-1M by Kumar Shivendu and Nirant Kasliwal (MIT license), and, for a smaller index, the first 100,000 rows. The rows are DBpedia entities embedded with OpenAI text-embedding-ada-002, 1,536 dimensions. The last 1,000 rows are the queries, so no row appears in both sets. The answer key is the exact top 10 nearest neighbours of every query, computed by brute force in 64-bit floating point independently of either engine, so both are graded against the same ground truth.

Engines and settings

Engines. HotKV commit c8dafd37 (VSIM.CREATEINDEX, VSIM.SET, VSIM.SEARCH), an unreleased development build compiled with default release flags, not -march=native. Redis 8.10.2 built from its release tag with a stock make (VADD, VSIM). Both build HNSW graphs.
Index settings. Cosine similarity, HNSW with M=16 and ef_construction=200 on both, set explicitly. They are also each engine's own defaults.
Precision. Full-precision float32 vectors on both. Redis defaults to 8-bit quantization, so it ran with NOQUANT.
Machine. One Azure Standard_D160alds_v7 (AMD EPYC 9V45, 160 vCPUs). Both servers were pinned to the same 128 vCPUs, the load generators to the other 32, all over loopback. HotKV ran 128 threads and Redis its default I/O threads. We tried 1, 4, 8 and 16 and the setting made no consistent difference.
Load generator. One process per client connection, each keeping one request outstanding, with the requests built before the timed part. The busiest load-generator process used 8% of one CPU in any run, so the client never set the limit.

Procedure

For search depths (ef) of 40, 80, 150 and 300, and 500 on the larger index, we ran the 1,000 queries repeatedly with 1, 2, 4, 8, 16, 32, 50 and 100 clients: one second of warm-up, then four seconds counted, three times each, with the order of the two engines alternating. We report the median. Recall was measured separately, over one connection. HotKV was loaded over 32 connections in every run. The Redis indexes were built once, over 32 connections with VADD ... CAS, saved, and loaded for the measurements. The scripts and every trial are in the benchmark repository.

Results

Matched recall targets

The two engines do not reach the same recall at the same ef, so we compare throughput at shared minimum recall targets: for each target, the fastest measured search depth that reaches it. The depths and recalls reached are in the tables, because they differ a little. This uses measured depths, not interpolation. Queries per second on 999,000 vectors:

Recall@10 target	Clients	HotKV	Redis 8.10.2	HotKV vs Redis	Median latency
95% or better	1	1,328 (ef 80, 0.970)	946 (ef 80, 0.962)	1.40x	0.77 / 1.07 ms
95% or better	8	10,342	5,531	1.87x	0.79 / 1.43 ms
95% or better	16	18,500	7,265	2.55x	0.87 / 2.15 ms
95% or better	50	37,508	6,825	5.50x	1.33 / 7.17 ms
95% or better	100	51,146	6,510	7.86x	1.91 / 13.49 ms
99% or better	1	507 (ef 300, 0.993)	287 (ef 500, 0.994)	1.77x	1.99 / 3.31 ms
99% or better	16	6,961	2,322	3.00x	2.30 / 6.70 ms
99% or better	50	12,393	2,483	4.99x	4.02 / 18.91 ms
99% or better	100	13,843	2,583	5.36x	7.11 / 37.65 ms

Median latency is HotKV / Redis. The same pattern held for a 90% target: 90,003 against 7,973 queries per second at 100 clients (11.3 times), both at ef 40.

Every search depth

ef	HotKV recall	Redis recall	HotKV, 1 client	Redis, 1 client	HotKV, 50 clients	Redis, 50 clients	HotKV, 100 clients	Redis, 100 clients
40	0.940	0.923	2,108	1,259	63,897	8,061	90,003	7,973
80	0.970	0.962	1,328	946	37,508	6,825	51,146	6,510
150	0.987	0.980	875	636	21,912	5,241	26,122	4,942
300	0.993	0.990	507	396	12,393	3,454	13,843	3,402
500	0.997	0.994	338	287	7,688	2,483	8,843	2,583

Queries per second on 999,000 vectors. On 100,000 vectors the pattern is the same. At 95% recall HotKV served 1.5 times Redis's queries per second with one client (1,816 against 1,214) and 8.9 times with 100 (63,247 against 7,119).

How much of this is noise

Each cell is the median of three trials. Half of the cells had a spread between their lowest and highest trial of 1.3% of the median or less, nine in ten 7% or less, and the widest was 26% (Redis, 100,000 vectors, ef 300, 50 clients). The report lists every trial.

Recall was measured once per index build. Three independent builds of each engine on 100,000 vectors differed by up to 0.013 at the same depth, so HotKV's higher recall is clear up to ef 300 and within build-to-build variation at ef 500 on the smaller index.

Loading and memory

Loaded over 32 connections each, HotKV took 188 seconds for the 999,000 vectors (5,325 per second) and Redis 935 seconds (1,069 per second), including the time the load generator spent formatting each vector. On 100,000 vectors it was 15 seconds against 57. Redis loaded faster with CAS than without in a check on 20,000 vectors (2,279 against 1,488 vectors per second), so we used it. The Redis load ran in an earlier step with Redis pinned to four vCPUs while other checks ran on the machine; HotKV loaded alone on 128 vCPUs. Treat the Redis loading time as a lower bound on its speed, not a like-for-like comparison.

After loading 999,000 vectors the HotKV process held 6.77 GB and the Redis process 7.01 GB.

What limits Redis 8.10.2 here

Redis vector sets run each search on its own thread, but the HNSW code has a fixed number of thread slots (32, set when Redis is compiled), and vectors are added on the main thread. In our runs, from 32 clients up, Redis kept between 6 and 30 of the machine's vCPUs busy and its throughput had levelled off. HotKV kept about 92 vCPUs busy at 100 clients on ef 80 and its throughput was still growing. That is how each engine behaves on this machine, not something the benchmark configured.

Limitations

These are HotKV Ltd's measurements of the versions and configuration above, not a claim about every Redis deployment.

The test setup

A development build. Commit c8dafd37 is a development build, not a release.
One dataset, one machine. 1,536-dimension OpenAI embeddings, top 10 only, float32, cosine. Other dimensions, data or hardware can give different ratios.
The client shares the machine. The load generators ran on separate vCPUs but shared the memory system and the last-level cache with the servers.
Persistence off. Neither server wrote snapshots or an append-only file.

What was not measured

Redis's 8-bit mode. Redis stores 8-bit quantized vectors by default, which uses less memory. We compared float32 against float32.
The Redis Query Engine. Redis also offers vector search through FT.CREATE and FT.SEARCH. We tested vector sets only, so the results say nothing about the Query Engine.
Redis loading and reload. The Redis indexes were built in an earlier run with Redis pinned to four vCPUs, saved, and reloaded for the measurements, so the Redis load times are a lower bound and are not comparable like for like with HotKV's. Searching a reloaded index was about 10 percent slower than a freshly built one in a check on 20,000 vectors, so Redis's query figures may be low by about that much.

Try it yourself

Vector search is built into the same server as HotKV's prompt cache, and the same vector index backs its semantic cache and agent memory. Vector search is included on every HotKV Cloud plan. The prompt cache, semantic cache and agent memory come with the Performance plan and above. Start on HotKV Cloud, or talk to sales to benchmark it against your own embeddings.

HotKV cut Claude token use by 70% in our Banking77 benchmark

HotKV Ltd — Mon, 19 Oct 2026 02:15:41 +0000

A semantic cache promises to cut LLM spend by answering repeated and similar questions from memory instead of calling the model. We measured that end to end: the same 3,000 customer support requests, sent to each of four Claude models twice, once straight to the model and once with HotKV in front, with Anthropic's own prompt caching left on in both runs. Here is exactly what we ran and what it found, including where it falls short.

In short. HotKV cut model calls, input plus output tokens and API cost by about 70% on each of the four models. Most of that comes from questions that repeat exactly. The semantic layer adds roughly 12 points on top. At the default 0.81 similarity threshold, cached answers were judged to fit the exact question 5.8 to 10.7 percentage points less often than fresh answers. On the one model we swept (Sonnet 5.5), a threshold of 0.88 or higher showed no such gap.

Updated 12 October 2026: this article now reports our larger run on four current Claude models. It replaces an earlier 1,500-request run on GPT and Claude Sonnet 5 that measured about 60% fewer tokens, which we no longer present.

What we tested

The workload

We replayed customer support questions from the test split of Banking77 (PolyAI, CC BY 4.0), a public dataset of short banking questions labelled with 77 intents. It was introduced in Casanueva et al., 2020. The 3,000 requests were drawn with a fixed seed. Intent popularity follows a Zipf distribution with exponent 1.0, a synthetic skew in which a few intents come up far more often than the rest, and the question within an intent is picked uniformly, with repeats. All 77 intents appear, and 1,264 of the 3,000 requests are distinct questions (counted by replaying the seeded request generator). The bank and its support policy are fictional. We wrote a policy of about 2,200 tokens, sent as the system prompt on every request, that asks for two to four short sentences.

Models and settings

We ran claude-fable-5-1, claude-opus-5-5, claude-sonnet-5-5 and claude-haiku-5-5. Fable 5.1 and Opus 5.5 ran at their lowest effort setting, because their thinking cannot be switched off. Sonnet 5.5 ran with up-front thinking off, its lowest setting, and Haiku 5.5 with thinking disabled. Maximum output was 2,000 tokens for Fable and Opus and 400 for Sonnet and Haiku, and every answer finished normally rather than being cut off. Eight requests ran at a time in both modes, calls were not streamed, and each condition ran once.

The two runs

Baseline: every request goes straight to the model, with Anthropic's prompt caching on (cache_control on the system prompt, default five-minute lifetime). This is the configured baseline used here, not a claim that every provider-side optimization was exhausted.
With HotKV: HotKV sits in front of the same traffic and the same system prompt, with the cache starting empty.

What we counted

Every token count comes from the usage block of the API response: uncached input, cache reads and writes, and output (thinking included). Embedding tokens are separate. Costs use Anthropic's list prices of 11 October 2026, include the embedding calls for semantic lookups, and exclude HotKV hosting and operations. They are not a quote for current pricing.

How the cache works

For each question, HotKV tries two lookups before the model is called:

Exact match: a GET on a SHA-256 key built from the model, the system prompt version and the normalized question (lower-cased, whitespace collapsed). It catches a question asked in identical words.
Semantic match: an SCACHE.GET call with an embedding of the question (OpenAI text-embedding-3-small, cosine similarity). It looks for a similar question. A similarity score alone cannot guarantee the cached answer fits.

Only a miss on both reaches the model, and its answer is written back with SET and SCACHE.SET. The similarity threshold for a semantic hit was fixed at 0.81 before the run. We chose it on a separate Banking77 train split as the lowest value, in steps of 0.01, whose semantic hits pointed at the same intent at least 98% of the time (98.2% there). In this run, on the untouched test split, 96.7% to 96.8% of semantic hits matched the intent of the question asked: 30 or 31 wrong-intent hits per model out of 926 to 936.

Results

Calls, tokens and cost

Model	Model calls	Tokens (input plus output)	API cost
Claude Fable 5.1	3,000 to 896	7.25M to 2.16M (70% lower)	$30.31 to $9.18 (70% lower)
Claude Opus 5.5	3,000 to 898	7.34M to 2.20M (70% lower)	$14.60 to $4.50 (69% lower)
Claude Sonnet 5.5	3,000 to 897	7.16M to 2.14M (70% lower)	$5.51 to $1.66 (70% lower)
Claude Haiku 5.5	3,000 to 897	7.08M to 2.11M (70% lower)	$0.27 to $0.08 (70% lower)

Across the four models, calls fell from 12,000 to 3,588, tokens by 70.1% and cost by 69.6%. About 70% of requests were answered from the cache on every model: roughly 39% by exact match and 31% by semantic match.

Calls, tokens and dollars fall together because HotKV removes whole calls. Read the token figure with care: 90% to 93% of the counted tokens are cache reads of the same 2,200-token policy prompt, which Anthropic bills at a small fraction of the normal input price. Output tokens are only 6% to 9% of the tokens but 73% to 92% of the baseline cost.

Speed

The median response time across the whole run fell from 2.7 seconds to 127 ms on Fable 5.1, from 2.9 seconds to 126 ms on Opus 5.5, from 1.8 seconds to 128 ms on Sonnet 5.5 and from 1.4 seconds to 125 ms on Haiku 5.5, which is 11 to 23 times faster. A repeated question came back in a median of about 0.05 ms, measured by a Python client over loopback on the same machine as HotKV, so it is a local figure rather than a network one. A paraphrased question, served from the semantic cache, took about 130 ms, almost all of it the embedding API call.

The whole-run median sits inside that semantic-hit band because exact hits (about 39% of requests) are too few to carry the median on their own, while exact and semantic hits together (70%) pass the halfway mark. The slow end moves less: 30% of requests still call the model. The 99th percentile was about 6.4 seconds with HotKV against 6.6 seconds without it on Fable 5.1, and about the same on Haiku 5.5. It was lower on Opus 5.5 (4.7 seconds against about 6) and on Sonnet 5.5 (2.7 seconds against 3.3). Calls were not streamed, so these are times to the complete answer, not to the first token.

Warm-up

The cache starts empty, so early requests still reach the model. The share of requests served from the cache was about 52% in the first thousand requests and about 81% in the last thousand.

Where the saving comes from

Most of the saving is plain repetition. An exact-match cache alone would have needed at most 1,264 model calls out of 3,000, one for each distinct question, which is about 58% fewer. The semantic layer took that to about 70%, so it contributes roughly 12 points on this workload. It is also the part that carries the answer-quality risk, as the next section shows. This is derived from the request sequence and the logs, not a separate measured run.

Answer quality

A semantic hit means the same intent, not the same correct answer. To measure correctness we had a different Claude model judge every semantic hit. The judge saw the customer's question and two answers in random order: the cached answer HotKV served, and the fresh answer the same model gave to that exact question in the baseline run. It said whether each answer correctly answered that exact question. Opus 5.5 judged the answers of Fable 5.1, Sonnet 5.5 and Haiku 5.5, and Fable 5.1 judged Opus 5.5, so no model graded its own output.

At the default threshold

Model	Semantic hits judged	Cached answer fits	Fresh answer fits	Gap
Claude Fable 5.1	926	83.3%	90.2%	6.9 points
Claude Opus 5.5	935	93.3%	99.0%	5.8 points
Claude Sonnet 5.5	936	77.9%	86.1%	8.2 points
Claude Haiku 5.5	936	67.9%	78.6%	10.7 points

At 0.81, a cached answer fit the exact question 5.8 to 10.7 percentage points less often than a fresh answer from the same model. Semantic hits are about 31% of requests, so across all responses, and counting exact hits as neutral, that is roughly 2 to 3 points (1.8 to 3.3). The judges differ in strictness, so compare cached with fresh only within one model.

Raising the threshold

We reran Sonnet 5.5 at four more thresholds, each from an empty cache, with the same judge. The baseline cost was $5.51.

Threshold	API cost	Lower by	Model calls	Semantic hits	Cached answer fits	Fresh answer fits
0.81	$1.66	70%	897	936	77.9%	86.1%
0.85	$1.88	66%	1,024	609	80.3%	85.6%
0.88	$1.99	64%	1,110	382	89.5%	83.8%
0.90	$2.13	61%	1,160	281	87.9%	86.5%
0.95	$2.22	60%	1,247	50	92.0%	90.0%

From 0.88 upward cached answers did not show a deficit against fresh ones, while still cutting cost by 64% at 0.88 and 61% at 0.90. The evidence is thin. At 0.90 the 95% interval for the difference runs from about 4 points worse to 7 points better, the 0.95 row has only 50 hits, and each row judges a different subset of hits. This is one model, one judge and one run per threshold, so it is not proof of equivalent quality or a universal safe threshold.

Near-identical questions

The real risk is questions that look almost identical but need different answers, such as depositing versus withdrawing. We had Opus 5.5 write 119 pairs, each a real training-split question plus a version with one decisive detail changed, and counted how many changed questions would clear the threshold against a cache holding only the original. These pairs are built to be hard, were not reviewed by a person, and show a worst case rather than how often it happens in real traffic.

Threshold	0.81	0.85	0.88	0.90	0.95
Changed questions that would match	62%	45%	30%	25%	10%
True paraphrases that would match	39%	23%	7%	4%	0%

Raising the threshold reduces the risk but does not remove it. Even at 0.90, 25% of the deliberately close pairs would match, and fewer true paraphrases match too.

What this means at scale

Extrapolating the measured cost per request to 1 million requests a day over a 30-day month, and assuming the same traffic pattern, gives these monthly figures:

claude-fable-5-1: about $303,100 without HotKV, $91,800 with it, roughly $211,300 saved.
claude-opus-5-5: about $146,000 without HotKV, $45,000 with it, roughly $101,000 saved.
claude-sonnet-5-5: about $55,100 without HotKV, $16,600 with it, roughly $38,500 saved.
claude-haiku-5-5: about $2,700 without HotKV, $800 with it, roughly $1,900 saved.

This is arithmetic on the measured cost per request, not another measured run. It excludes hosting, and the capacity needed for that volume was not measured. At the 3,000 requests we ran, the saving is only $0.19 for Haiku 5.5 and $21.12 for Fable 5.1. The four models ran with short answers, and Fable 5.1 and Opus 5.5 at their lowest effort, so longer or reasoning-heavy answers would make each avoided call worth more.

Limitations

This is not a general "HotKV saves you 70%" claim. It is what we measured on this workload, and the workload does the heavy lifting.

Savings track repetition. With every intent equally likely instead of Zipf-skewed, cost fell 56% on Sonnet 5.5 and Haiku 5.5. With 3,000 all-different questions it fell 38%, entirely from semantic matches between different questions of the same intent, which Banking77 has plenty of. A workload with no repeated or similar questions saves nothing.
Synthetic traffic. Draws are independent: no bursts, sessions or multi-turn context. One public dataset of short banking questions, a fictional policy written for this test, and a finite pool of about 3,080 test questions.
A model judges the answers. No human reviewed them, there was no second judge, and every judge is a Claude model. Fresh answers failed the judge's "fits" test between 1% and 21% of the time, so it is a strict bar.
Same intent is not same answer. The 0.81 threshold was calibrated on intent labels, not answer quality. The threshold sweep and the near-identical test cover one model and one embedding model.
Small test rig. One run per condition, a single four-vCPU server with the client on the same machine, and a cache that was not warmed before the HotKV runs.
Costs are API costs. They exclude HotKV hosting and operations. Validate answer quality on your own questions, and use exact caching or fresh answers where semantic reuse is unsuitable.

The per-model figures are also on the AI page.

Try it yourself

The exact-match cache in this test uses plain GET and SET, which every HotKV plan includes. The semantic cache (SCACHE.*) comes with the Performance plan and above (pricing). Start on HotKV Cloud and point your existing traffic at it, or talk to sales if you want to work through the numbers for your own workload first.

Originally published at hotkv.com.