Designing an application with Redis as a data store. What? Why?

1) Introduction

Hello everyone! Many people know what Redis is, and if you don’t know, the official site can bring you up to date.
For most Redis is a cache and sometimes a message queue.
But what if we go a little crazy and try to design an entire application using only Redis as data storage? What tasks can we solve with Redis?
We will try to answer these questions, in this article.

What we won't see here?

• Every Redis data structure in detail won't be here. For what purposes you should read special articles or documentation.
• Here will also be no production-ready code that you could use in your work.

What we'll see here?

• We'll use various Redis data structures to implement different tasks of a dating application.
• Here will be Kotlin + Spring Boot code examples.

2) Learn to create and query user profiles.

• For the first, let's learn how to create user profiles with their names, likes, etc.

  To do this, we need a simple key-value store. How to do it?

• Simply. A Redis has a data structure - a hash. In essence, this is just a familiar hash map for all of us.

Redis query language commands can be found here and here.
The documentation even has an interactive window to execute these commands right on the page. And the whole commands list can be found here.
Similar links work for all subsequent commands that we will consider.

In the code, we use RedisTemplate almost everywhere. This is a basic thing for working with Redis in the Spring ecosystem.

The one difference from the map here is that we pass "field" as the first argument. The “field” is our hash's name.

fun addUser(user: User) {
        val hashOps: HashOperations<String, String, User> = userRedisTemplate.opsForHash()
        hashOps.put(Constants.USERS, user.name, user)
    }

fun getUser(userId: String): User {
        val userOps: HashOperations<String, String, User> = userRedisTemplate.opsForHash()
        return userOps.get(Constants.USERS, userId)?: throw NotFoundException("Not found user by $userId")
    }

Above is an example of how it might look in Kotlin using Spring's libraries.

All pieces of code from that article you can find on Github.

3) Updating user likes using Redis lists.

• Great!. We have users and information about likes.

  Now we should find a way how to update that likes.

  We assume events can happen very often. So let's use an asynchronous approach with some queue. And we will read the information from the queue on a schedule.

• Redis has a list data structure with such a set of commands. You can use Redis lists both as a FIFO queue and as a LIFO stack.

In Spring we use the same approach getting ListOperations from RedisTemplate.

We have to write to the right. Because here we are simulating a FIFO queue from right to left.

fun putUserLike(userFrom: String, userTo: String, like: Boolean) {
        val userLike = UserLike(userFrom, userTo, like)
        val listOps: ListOperations<String, UserLike> = userLikeRedisTemplate.opsForList()
        listOps.rightPush(Constants.USER_LIKES, userLike)
}

Now we're going to run our job on schedule.

We are simply transferring information from one Redis data structure to another. This is enough for us as an example.

fun processUserLikes() {
        val userLikes = getUserLikesLast(USERS_BATCH_LIMIT).filter{ it.isLike}
        userLikes.forEach{updateUserLike(it)}
}

User updating is really easy here. Give a hi to HashOperation from the previous part.

private fun updateUserLike(userLike: UserLike) {
        val userOps: HashOperations<String, String, User> = userLikeRedisTemplate.opsForHash()
        val fromUser = userOps.get(Constants.USERS, userLike.fromUserId)?: throw UserNotFoundException(userLike.fromUserId)
        fromUser.fromLikes.add(userLike)
        val toUser = userOps.get(Constants.USERS, userLike.toUserId)?: throw UserNotFoundException(userLike.toUserId)
        toUser.fromLikes.add(userLike)

        userOps.putAll(Constants.USERS, mapOf(userLike.fromUserId to fromUser, userLike.toUserId to toUser))
    }

And now we show how to get data from the list. We are getting that from the left. To get a bunch of data from the list we will use a range method.
And there is an important point. The range method will only get data from the list, but not delete it.

So we have to use another method to delete data. trim do it. (And you can have some questions there).

private fun getUserLikesLast(number: Long): List<UserLike> {
        val listOps: ListOperations<String, UserLike> = userLikeRedisTemplate.opsForList()
        return (listOps.range(Constants.USER_LIKES, 0, number)?:mutableListOf()).filterIsInstance(UserLike::class.java)
            .also{
listOps.trim(Constants.USER_LIKES, number, -1)
}
}

And the questions are:

• How to get data from the list into several threads?
• And how to ensure the data won't lose in case of error? From the box - nothing. You have to get data from the list in one thread. And you have to handle all the nuances that arise on your own.

4) Sending push notifications to users using pub/sub

• Keep moving forward!
  We already have user profiles. We figured out how to handle the stream of likes from these users.

  But imagine the case when you wanna send a push notification to a user the moment we got a like.
  What are you gonna do?

• We already have an asynchronous process for handling likes, so let's just build sending push notifications into there. We will use WebSocket for that purpose, of course. And we can just send it via WebSocket where we get a like. But what if we wanna execute long-running code before sending? Or what if we wanna delegate work with WebSocket to another component?
• We will take and transfer our data again from one Redis data structure (list) to another (pub/sub).

fun processUserLikes() {
        val userLikes = getUserLikesLast(USERS_BATCH_LIMIT).filter{ it.isLike}
                pushLikesToUsers(userLikes)
        userLikes.forEach{updateUserLike(it)}
}

private fun pushLikesToUsers(userLikes: List<UserLike>) {
  GlobalScope.launch(Dispatchers.IO){
        userLikes.forEach {
            pushProducer.publish(it)
        }
  }
}

@Component
class PushProducer(val redisTemplate: RedisTemplate<String, String>, val pushTopic: ChannelTopic, val objectMapper: ObjectMapper) {

    fun publish(userLike: UserLike) {
        redisTemplate.convertAndSend(pushTopic.topic, objectMapper.writeValueAsString(userLike))
    }
}

The listener binding to the topic is located in the configuration.
Now, we can just take our listener into a separate service.

@Component
class PushListener(val objectMapper: ObjectMapper): MessageListener {
    private val log = KotlinLogging.logger {}

    override fun onMessage(userLikeMessage: Message, pattern: ByteArray?) {
        // websocket functionality would be here
        log.info("Received: ${objectMapper.readValue(userLikeMessage.body, UserLike::class.java)}")
    }
}

5) Finding the nearest users through geo operations.

• We are done with likes. But what about the ability to find the closest users to a given point.

• GeoOperations will help us with this. We will store the key-value pairs, but now our value is user coordinate. To find we will use the [radius](https://redis.io/commands/georadius) method. We pass the user id to find and the search radius itself.

Redis return result including our user id.

fun getNearUserIds(userId: String, distance: Double = 1000.0): List<String> {
    val geoOps: GeoOperations<String, String> = stringRedisTemplate.opsForGeo()
    return geoOps.radius(USER_GEO_POINT, userId, Distance(distance, RedisGeoCommands.DistanceUnit.KILOMETERS))
        ?.content?.map{ it.content.name}?.filter{ it!= userId}?:listOf()
}

6) Updating the location of users through streams

• We implemented almost everything that we need. But now we have again a situation when we have to update data that could modify quickly.

  So we have to use a queue again, but it would be nice to have something more scalable.

• Redis streams can help to solve this problem.
• Probably you know about Kafka and probably you even know about Kafka streams, but it isn't the same as Redis streams. But Kafka itself is a quite similar thing as Redis streams. It is also a log ahead data structure that has consumer group and offset. This is a more complex data structure, but it allows us to get data in parallel and using a reactive approach.

See the Redis stream documentation for details.

Spring has ReactiveRedisTemplate and RedisTemplate for working with Redis data structures. It would be more convenient for us to use RedisTemplate to write the value and ReactiveRedisTemplate for reading. If we talk about streams. But in such cases, nothing will work.
If someone knows why it works this way, because of Spring or Redis, write in the comments.

fun publishUserPoint(userPoint: UserPoint) {
    val userPointRecord = ObjectRecord.create(USER_GEO_STREAM_NAME, userPoint)
    reactiveRedisTemplate
        .opsForStream<String, Any>()
        .add(userPointRecord)
        .subscribe{println("Send RecordId: $it")}
}

Our listener method will look like this:

@Service
class UserPointsConsumer(
    private val userGeoService: UserGeoService
): StreamListener<String, ObjectRecord<String, UserPoint>> {

    override fun onMessage(record: ObjectRecord<String, UserPoint>) {
        userGeoService.addUserPoint(record.value)
    }
}

We just move our data into a geo data structure.

7) Count unique sessions using HyperLogLog.

• And finally, let's imagine that we need to calculate how many users have entered the application per day.
• Moreover, let's keep in mind we can have a lot of users. So, a simple option using a hash map is not suitable for us because it will consume too much memory. How can we do this using fewer resources?
• A probabilistic data structure HyperLogLog comes into play there. You can read more about it on the Wikipedia page. A key feature is that this data structure allows us to solve the problem using significantly less memory than the option with a hash map.

fun uniqueActivitiesPerDay(): Long {
    val hyperLogLogOps: HyperLogLogOperations<String, String> = stringRedisTemplate.opsForHyperLogLog()
    return hyperLogLogOps.size(Constants.TODAY_ACTIVITIES)
}

fun userOpenApp(userId: String): Long {
    val hyperLogLogOps: HyperLogLogOperations<String, String> = stringRedisTemplate.opsForHyperLogLog()
    return hyperLogLogOps.add(Constants.TODAY_ACTIVITIES, userId)
}

8) Conclusion

In this article, we looked at the various Redis data structures. Including not so popular geo operations and HyperLogLog.
We used them to solve real problems.

We almost designed Tinder, it is possible in FAANG after this)))
Also, we highlighted the main nuances and problems that can be encountered when working with Redis.

Redis is a very functional data storage. And if you already have it in your infrastructure, it can be worth looking at Redis as a tool to solve your other tasks with that without unnecessary complications.

PS:
All code examples can be found on github.

Write in the comments if you notice a mistake.
Leave a comment below about such a way to describe using some technology. Do you like it or not?

And follow me at Twitter:🐦@de____ro