Redis is the most popular key-value database store. Redis is used in the majority of large scale applications.
While building large scale distributed systems, we often end up in a scenario where we have to perform the huge amount of data ingestion or data transfer from one system to another. The only feasible way to perform this is through streaming. Distributed engineers generally use streaming software such as Apache Kafka to perform data streaming and ingestion.
Redis with the support of version 5.0 offers an innovative way to manage streams. Redis Stream is a built-in feature and it offers a data structure that developers can use to perform data ingestion, data consumption, creates a data channel between producer and consumers, and so on.
In this tutorial, we are going to explore how to use Redis streams and how you as a distributed systems engineer can incorporate Redis streams in your architecture.
Redis streams capabilities
Redis is used in various use cases across the system. This includes content caching, real-time analytics, message brokers, etc.
While Redis already provides a publish/subscribe feature, with the release of Redis streams, Redis offers the following capabilities:
- Collect large amounts of data.
- Create a data channel between many consumers and producers.
- Data ordering is effectively managed even though the rate of data ingestion and data consumption is different.
- Offline data persistence support.
- Asynchronous communication between producer and consumer.
- Easily scalable to large number producers and consumers.
Like I mentioned earlier, Redis streams are built-in into Redis. You don’t need to perform extra installation or management of heavy software. It’s right into Redis.
Let’s jump right into Redis streams and learn how to use it.
Using Redis Streams
Redis Streams is an append-only log-based data structure. Redis streams offer commands to add data in streams, consume streams, and manage how data is consumed.
Redis streams can have one to one communication or one to many or many to many communication streams between producers and consumers. Redis streams also offer consumer groups so that they can consume only the information that they are supposed to receive.
Let’s use Redis streams commands. I am assuming that you have Redis version 5.0 or above already installed in your system. If not, please navigate to the Redis download page to download the latest version of Redis.
Once you have Redis installation sorted out, start the Redis server if you are using it in your system using the following command.
redis-server
Some installers already run the Redis server in the background, so if you get some error, don’t worry about it.
Now, let’s start the Redis command line shell using the following command.
redis-cli
Let’s add some data in Redis streams. The default and easy way are to run this command.
XADD mystream * name Shahid
Here, mystream is the name of the stream. Name is the key and Shahid (My name) is the value.
When you run this command, Redis will generate the ID for your stream data.
We can also add our own ID while adding the records. For example,
XADD mystream 10000000 name Shahid
However, you should rely on a system-generated unique ID.
Let’s consume data from the Redis streams.
To read everything from the start of the stream to the end, we can use the following command.
XREAD STREAMS mystream 0
Here we are giving the name of the stream and start index i.e 0. This command will return all the records present in the stream.
We can also limit the records by providing the record count.
XREAD COUNT 100 STREAMS mystream 0
We can also iterate through the stream using the XRANGE command.
For example,
XRANGE mystream 1518951123450-0 1518951123460-0
The command shown above will read the record from the stream between the ID mentioned in the command.
If we don’t know the ID of the records, we can pick the number of records using the COUNT operator.
XRANGE mystream - + COUNT 10
The command shown above will return the record from 10 records iterating from the beginning of the stream.
Redis streams offer more commands and ways to manage streams. You can learn more about it on the documentation page here.
Top comments (3)
Neat, thanks for the article. It feels like Redis streams are similar to Kafka partition. Is that right?
Yes, kind of. I believe they took inspiration from Kafka itself.
Makes sense. Thanks Shahid!