Using Redis for Caching (2023)

In case it helped :)

We will cover briefly:

1. Caching APIs (Go) using Redis
2. Redis Subscription using Pub/Sub
3. Calling API using React

What is Redis?

Redis, which stands for Remote Dictionary Server, is a fast, open source, in-memory, key-value data store. It delivers sub-millisecond response times, enabling millions of requests per second for real-time applications.

Redis is a key-value-based NoSQL database that stores data in memory, i.e. in RAM.

Use cases of Redis

• Caching
• Geospatial
• Chat, messaging, and queues
• Gaming leaderboards

Advantages of Redis

• Simple, fast, and easy to use
• Supports a variety of data structures
• Allows storing key and value pairs as large as 512 MB 
• High availability using Redis Sentinel

Disadvantages of Redis

• Requires huge RAM (although depends on the type of application)
• Failover happens if the master at least 1 slave
• Data can only be accessed via keys

Setting up Redis 

We will setup Redis using Docker

docker run -d -p 6379:6379 --name redis redis

Nowadays there are some hosting providers such as AWS, Redis Labs, or Upstash that provide Redis on the cloud.

There are many ways of connecting to Redis. In our case, we will be using the RedisCLI Since we have the docker setup ready, let's connect.

Here is an example of using the RedisCLI tool to set, get and delete a value from the Redis

% docker exec -it redis redis-cli
127.0.0.1:6379> set key value
OK
127.0.0.1:6379> get key
"value"
127.0.0.1:6379> del key
(integer) 1
127.0.0.1:6379>

Some other useful commands

keys *: for finding all the keys

FLUSHALL : delete all keys from all databases.
FLUSHDB : delete all keys from the currently selected DB.

EXPIRE key 120: key will be deleted in 120seconds

Caching APIs (Go) using Redis

There are multiple clients available in Go for implementing Redis. However, in this article, we will be using Go-redis

Go-Redis

• Go-Redis is a type-safe, Redis client library for Go. 
• It is a Redis client able to support a Redis cluster and is designed to store and update slot info automatically with a cluster change.
• It supports features like Pub/Sub Sentinel, and pipelining

Create APIs using Gorilla Mux

• We will be using Gorilla Mux to create the APIs locally
• It implements a request router and dispatcher to match the incoming requests.

Install it using

go get -u github.com/gorilla/mux

• We will register the following endpoints:

GET /users/:id — to get a user’s information based on the id. Sample response

[
   {
      "id":"116c24b1-9425-4fe4-aec2-86ba7384733e",
      "name":"Bob",
      "age":29,     
      "source":""
   },
   {
      "id":"a3fa13d4-67a9-4ae6-9713-dce7043844d7",
      "name":"Alice",
      "age":29,
      "source":""
   }
]

GET /users — returns the users present in the database. Sample response

 

{
   "id":"116c24b1-9425-4fe4-aec2-86ba7384733e",
   "name":"Bob",
   "age":29,
   "source":""
}

• Next, we create the router instance using the mux.NewRouter() and assign the above-created routes to the respective handlers
• Each of the corresponding handlers is passed ResponseWriter and Request as parameters, which help in returning the desired response to the client.
• We then specify the server details using http.Server for running the HTTP server

srv := &http.Server{
 Handler:      router,
 Addr:         ":8081",
 WriteTimeout: 15 * time.Second,
 ReadTimeout:  15 * time.Second,
}
log.Fatal(srv.ListenAndServe())

Handler : This is the object that responds to the incoming HTTP requests (which we created above)

Addr : This specifies the TCP address for the server to listen on, which by default is 80 

WriteTimeout : Maximum duration before timing out the writes of the response

ReadTimeout : Maximum duration for reading the entire incoming request

• Finally, we run the server using ListenAndServe which listens on the network address specified in the Addr and serves the requests based on the Handler 

Setting up Postgres

We will be using Postgres for our database. Install it using docker with the following

docker run \
  -d \                                     
  -e POSTGRES_HOST_AUTH_METHOD=trust \
  -e POSTGRES_USER=user \
  -e POSTGRES_PASSWORD=password \
  -e POSTGRES_DB=dbname \
  -p 5432:5432 \
  postgres

• Verify if the new container is created and running at 0.0.0.0:5432docker ps -a For managing the database from the browser, install pgAdmin and connect to it using the above credentials, if all is good, you should see

Postgres database

We will create a table called users which will have the following schema

Users table 

Let’s insert some dummy data using 

INSERT INTO public.users(
 id, created_time, name, updated_time, age)
 VALUES (uuid_generate_v4(), NOW(), 'alice', NOW(), 29);
// IN CASE ANY ERROR RUN THIS
CREATE EXTENSION IF NOT EXISTS "uuid-ossp"

For connecting to Postgres using Go we will install this Here is the snippet

host := "127.0.0.1"
port := "5432"
user := "user"
password := "password"
dbname := "dbname"
psqlInfo := fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=disable", host, port, user, password, dbname)
result, err := sql.Open("postgres", psqlInfo)
if err != nil {  
   log.Fatalf("Error in connection : %s", err)
}

Here is the struct for the users 

type Users struct {
 ID          uuid.UUID `json:"id"`
 Name        string    `json:"name"`
 Age         int       `json:"age"`
 CreatedTime string    `json:"created_time"`
 UpdatedTime string    `json:"updated_time"`
 Source      string    `json:"source"`
}

API and get users by id API are created, we will use the Gorilla Mux to create a localhost server and listen to the endpoints

Caching APIs

Redis between DB and client

For connecting to the Redis we create a Redis Client using NewClient We specify the address at which the Redis exists

client := redis.NewClient(&redis.Options{
 Addr:        "127.0.0.1:6379",
 DB:          0,
 DialTimeout: 100 * time.Millisecond,
 ReadTimeout: 100 * time.Millisecond,
})

• The configuration options are available through the redis.Options parameter.

Addr : String of host and the port address, since we are hosting Redis locally, the value is 127.0.0.1 and by default, Redis runs on the port 6379 

DB : The database which will the selected after connecting to the server. By choosing 0 we mean to use the default database.

DialTimeout : In case our connection to the Redis server gets broken, we specify the timeout for establishing the new connection

ReadTimeout : This allows putting a timeout for the socket reads. In case any of the Redis server requests reach this timeout, the command calling it will fail instead of blocking the server.

• To check if we connected to the server, we call Ping using the client we created above. If there is no error, that implies we are connected to the Redis server.
• Finally, we return the Redis client which internally may have zero or more connections.

if _, err := client.Ping().Result(); err != nil {
 return nil, err
}
return &Client{
 client: client,
}, nil

Get and Set Keys

• The use case we would be following is whenever the front end asks for the details of a particular user, we would fetch it from the API and then cache it. 
• Subsequent requests for the particular user would be served from the cache until the cache key expires (which we set to 20 seconds )
• We will be making use of Set to set the value in the cache

func (c *Client) SetUser(key string, user structs.Users) {
 json, err := json.Marshal(user)
 if err != nil {
  log.Fatal(err)
 }
 c.client.Set(key, json, 20*time.Second)
}

Note: Here we are taking the userstruct as input. We are then converting the Go struct into JSON (aka marshaling) since JSON is a language-independent data format.

• We are setting the key pair with an optional expiration parameter of 20 seconds This means the key will automatically expire in the given duration.
• In case there is no expiration parameter, it means Zero expiration meaning the key has no expiration time.

Get keys

• We will be making use of Get for retrieving the value of the key

func (c *Client) GetUser(key string) (user *structs.Users) {
 val, err := c.client.Get(key).Result()
 resp := structs.Users{}
 err = json.Unmarshal([]byte(val), &resp)
 return &resp
}

• The response is in the form of a string and then we convert the byte data into the original user struct (aka unmarshalling)
• Inside this function, we additionally add a Redis publisher (which we will see in the next section).

Calling APIs

We have this API endpoint hosted locally GET /users/:id — to get a user’s information based on the id.

• Once the client calls this endpoint, we first check in case the value for this request can be served from the cache.

Note: In case the data exists inside the cache, we set the source: "cache"

• Logically, the first request will always hit the server, hence the time taken for getting the response will be more.

Note: For getting the value from the server, we set the source: "API"

router.HandleFunc("/users/{id}", func(w http.ResponseWriter, r *http.Request) {
 id := mux.Vars(r)["id"]
 val := redis.GetUser(id)
 if val != nil {
  val.Source = "cache"
  renderJSON(w, &val, http.StatusOK)
  return
 }
 user, err := GetUserByID(id)
 if err != nil {
  renderJSON(w, &user, http.StatusOK)
  return
 }
 redis.SetUser(id, user)
 user.Source = "API"
 renderJSON(w, &user, http.StatusOK)
})

• As we can see in the above snippet, we first invoke redis.GetUser to check the cache.
• This function checks for the id inside the cache, if the id is present, it returns the value.
• If the result fails, the function returns null and we proceed to invoke the API GetUserByID which hits our Postgres database

{
   "id":"116c24b1-9425-4fe4-aec2-86ba7384733e",
   "name":"Bob",
   "age":29,
   "source":"API"
}

s how it looks when the key is not present in Redis

Key not present in Redis

• The result from the API is then cached in Redis, so the subsequent requests are now served by the cache (until 20 seconds which is the key expiration duration)

{
   "id":"116c24b1-9425-4fe4-aec2-86ba7384733e",
   "name":"Bob",
   "age":29,
   "source":"cache"
}

This is how it looks when the key is present in Redis

Key present in Redis

---

Redis Subscription using Pub/Sub

• In the field of software engineering, publish-subscribe is a pattern where senders (publishers) categorize the messages into channels without knowing if there may be any subscribers.
• On the other hand, subscribers show interest in one or more channels and only receive messages that are of interest, without knowing if there may be any publishers.
• This decoupling of publishers and subscribers enhances greater scalability.

Redis Pub/Sub

• We will be using Redis for pub/sub, however, there are various other alternatives like Apache Kafka, Google cloud Pub/Sub, etc
• In order to subscribe to channels, for instance, foo and bar the client uses SUBSCRIBE providing the names of the channels:

SUBSCRIBE foo bar

Messages sent by other clients to these channels will be pushed by Redis to all the subscribed clients.

• Once we have a Redis client subscribing to a channel, that client can no longer execute any other operations except unsubscribing from the current channel or subscribing to more channels.

Go-Redis Publisher

Go-redis allows to publish messages and subscribe to channels. It also automatically re-connects to Redis Server when there is a network error.

• Inside our get user function, we will create a publisher, using redis.Publish(). This function takes two arguments, the name of the channel to which we want to send the message and the message.
• In our case, we set the channel name to send-user-name and the payload is the response from redis.GetUser (see above)
• We are sending the marshaled payload to the channel using Publish This is because it allows transferring the data as []byte 
• Since we are using a User struct, it can be encoded into a []byte 

// Sample payload we are sending using pub/sub
{
   "id":"116c24b1-9425-4fe4-aec2-86ba7384733e",
   "name":"Bob",
   "age":29,
   "source":"cache"
}
// Publish using Redis PubSub
payload, err := json.Marshal(resp)
if err := c.client.Publish("send-user-name", payload).Err(); err != nil {
 log.Fatal(err)
}

Go-Redis Subscriber

• We create a subscriber using the redis.Subscribe() function. 
• We will have one argument, the channel we want to subscribe to, which in our case is send-user-name.
• Subscribe subscribes to the client to the specified channels. Since this method does not wait on a response from Redis, so the subscription may not be active immediately.
• We create a new file that connects to the same Redis instance and calls the following

topic := redisClient.Subscribe("send-user-name")
channel := topic.Channel()
for msg := range channel {
 u := &User{}
 // Unmarshal the data into the user
 err := u.UnmarshalBinary([]byte(msg.Payload))
 if err != nil {
  panic(err)
 }
fmt.Printf("User: %v having age: %v and id: %v\n", u.Name, u.Age, u.ID)
 log.Println("Received message from " + msg.Channel + " channel.")
}

• The result of the subscribe is a PubSub and we extract the channel property out of this object
• This Channel returns a Go channel for concurrently receiving messages. If the channel is full for 30 seconds the message is dropped.
• We loop over the channel and extract the data from it, which is of the type Message Each message object comprises of

Channel : This is the channel name

Payload : The marshaled data received in the channel

Pattern : Pattern for the message

• The data received inside the message can be fetched using msg.Payload Since its marshaled, we will unmarshal it using the UnmarshalBinary and transform back into User struct.
• Finally, we can access the struct properties and print inside the console. We also print the message’s channel name, just for fun!!

Redis subscriber

Here, we get the message channel name send-user-name and the user details for Bob 

Calling API using React

In the above section, we created and hosted the APIs locally using Gorilla Mux. We will now consume these APIs from the front end using React

We will be using Axios for calling the APIs. Let’s install this dependency. 

npm install axios

• Axios is a lightweight HTTP client which makes calling requests very intuitive. It is similar to the JavaScript Fetch API 
• It works well with JSON data and does the heavy lifting of setting the request headers.
• It has better error handling for accessing the response and integrates well with async-await syntax.
• It can be used on the server as well as the client.

Integrate into React

We will be using the hooks useEffect and useState A hookis a special function that lets you “hook into” React features.

useEffect

• Using this Hook, we instruct React that our components need to do something after rendering. By default, useEffect runs after each render of the component where it’s called.
• A common use of this hook is to fetch data and display it.
• React remembers the function you passed, and invokes it later after the DOM updates.

Note: We have an empty array at the end of the useEffect Hook for making sure the internal function inside renders only once.

• If we want our effects to run less often, we provide the second argument (which is an array of values). These can be considered as the dependencies for the effect and if any changes (since the last time), the effect runs again.

useState

• It is a way of adding a state to the components and preservesome values between the function call. 
• In general, variables “disappear” when the function exits but state variables are preserved by React.
• This state can be an array, object, number, boolean, etc.
• The argument passed to the useState() is the initial state. 

const [fetchedData, setFetchedData] = useState([]);
useEffect(() => {
  const getData = async () => {
      const data = await axios.get(
          "http://localhost:8081/users"
      );
      setFetchedData(data);
  };
  getData();
}, []);

• To make a GET request using Axios we use the .get() method. The response is returned as an object.
• We use the .data property from the response to get the requested data and set it inside the hook using the setFetchedData 

Request responses:

When the page loads for the first time, the API hits our backend. The result fetchedData is then displayed on the page. 

Notice, source: API 

Source: API

And on the next request, the API hits our cache, displayed with source:cache

Source: Cache

Source code for Go

Source code for React

In case it helped :)