This article is a part of the blog below, go to original version for a full read.
https://www.alibabacloud.com/blog/a-detailed-explanation-of-the-detection-and-processing-of-bigkey-and-hotkey-in-redis_598143
Definition of BigKey and HotKey
We can often see the definitions of BigKey and HotKey in Guideline for Developing With Redis in companies, or in numerous articles about Redis best practices on the Internet. Even though the criteria for judging BigKey and HotKey are different, it is clear that the dimensions of judging are the same. BigKey is usually determined by the data size and number of members, while HotKey is determined by the frequency and number of requests Redis receives.
Definition of BigKey
Generally, we call a key that contains large size of data or a large number of members and lists as BigKey. There are some examples to help you fully understand the characteristics of BigKey, find as below.
- A STRING key whose size of value is 5 MB (the data is too large).
- A LIST key with 20,000 elements (the number of elements in the list is excessive).
- A ZSET key with 10,000 members (the number of members is excessive).
- A HASH key whose size is 100MB even if only contains 1,000 members (the size of the key is too large). It should be noted that the definition of BigKey might be different according to actual use cases and business scenarios of Redis. This is to say that you should judge taking all factors into consideration. In given examples, you can see some specific numbers for the size, members, and elements in a key, which is not a common definition of a BigKey, it's only to simplify the explanation, cannot be regarded as a standard in fact.
Definition of HotKey
When the workload of access to a key is significantly higher than that of other keys, we can call it a HotKey. To help you better understand what HotKeys looks like, here are some examples, please check as below.
- The total workload of access per second of a Redis instance is 10,000, while that of one key reaches 7,000. (Obviously, this key is a HotKey whose access is significantly higher than that of other keys.)
- A HASH key with thousands of members and a total size of 1MB receives a large amount of HGETALL requests each second. (In this case, we call it a HotKey since accesses to one key cost significantly more bandwidth than that of other keys.)
- A ZSET key with tens of thousands of members receives a large number of ZRANGE requests each second. (It's clear that the CPU run time is significantly higher than that spent on requests of other keys. Similarly, we can say this kind of key with large CPU consuming is a HotKey.)
Problems Caused by BigKey and HotKey
When using Redis, BigKey and HotKey bring various problems. The most common ones are performance degradation, access timeout, access skew and data skew.
Typical Problems Caused by BigKey
- Clients get delayed responses and it feels like the Redis is slowing down.
- The Redis memory usage keeps growing which triggers OOM, or write blocking and eviction of important keys occurs because of reaching the maxmemory.
- Access skew occurs which may cause a Redis instance to reach the performance bottleneck, so that the whole cluster also reaches the performance bottleneck. In this case, the memory usage of one node in Redis Cluster usually far exceeds that of other nodes because of access requsts for BigKeys on it, and the minimum granularity of data migration in Redis Cluster is key that the memory on that node cannot be balanced. This is to say the problem cannot be resolved unless you find a method to divide BigKeys into small keys.
- All services the Redis should be offering are affected as Redis is getting slower, which is all because read requests on a BigKey occupy all the bandwidth of Redis.
- Synchronization interruption or failover occurs because of long time blocking of primary database when deleting a BigKey.
Typical Problems Caused by HotKey
- HotKeys usually have long CPU run time, which deteriorates Redis performance and affects other requests.
- Hotspot on some Redis nodes/machines, instead of on keys that sharding to different Redis nodes, often prevent you from taking full advantage of your Redis Cluster. As the memory/cpu load of these nodes will be quite heavy, while other nodes are not efficiently used.
- As for panic buying and flash sales scenarios, oversold often occurs because the number of read requests for inventory on keys of corresponding goods is too much beyond the instance performance that the Redis cannot handle.
- The traffic on HotKeys suddenly surges to the maximum threshold the Redis can tolerate, even causing the cache service to crash, which is commonly referred to as a Redis avalanche. If this is the case, a large number of requests will directly hit backend database, causing a high load to the database layer that the database may be unable to withstand and downtime, thus affecting the business.
Causes of BigKey and HotKey
Insufficient business planning, incorrect use of Redis, accumulation of invalid data, and sudden increase in access can cause BigKey and HotKey issues. For example:
1) BigKey: Use Redis for inappropriate data types can introduce BigKey problems. For example, using String keys to store large binary files will make the values of the keys too large.
2) BigKey: Insufficient planning and design before business launch and no proper sharding policies or split plans to divide members in an individual key to multiple keys, resulting in an excessive number of members in a particular key.
3) BigKey: No regular cleanup on invalid data in HASH keys, resulting in a continuous increase of members in HASH keys, which may bring BigKey problems.
4) HotKey: Unexpected increase of access traffic owing to hot products, hot news, KOL (Key Opinion Leader) live streaming event or an online games battle that invloves a large number of players.
5) BigKey: Logical failures on business side that prevents LIST keys from being consumed, which result in an increasing number of members in the corresponding Key with no trend to decrease.
Next, let's see how to discover BigKey and HotKey in Redis.
↓
Click here to continue your read.
Top comments (0)