DEV Community: Sorin Curescu

Advanced Discord.js: Cache APIs Requests with Redis

Sorin Curescu — Fri, 23 Apr 2021 09:00:00 +0000

Getting started
- Installation
- Usage
Cache data from APIs
- Preparation
- Solving the problem
  - Implementation
Wrapping up
- Real-world usecases

This article is part of a series that can be found here. We're reusing some code from the previous article

Looking for a Discord.JS bot? Check out Hans! Now it's open-sourced 🥳!

Nowadays we depend on many APIs that we interact with (weather, games stats, etc...)

Many times we don't have to worry about rate limits but in some cases, we do.
If the API has a low rate limit (e.g. x amount of requests per minute) and if we want to deliver the maximum amount of data to our users at some point caching could be the best way to do it.

NOTE: The other option would be to store it in a persistent database such as MongoDB or SQLite, but this would be slower and we add extra network requests if we use an external service

Getting started

First of all, what is Redis?

Redis is an in-memory data structure store, used as a distributed, in-memory key–value database, cache and message broker, with optional durability. - redis.com

This looks promising!

It's storing data in memory so it will be amazingly fast to read/write.
We can temporally store data (it can also be persistent). For us we're interested in temporary caching, we don't want to show outdated data.

NOTE: You can check out the guide for MacOS or Windows. More information is available at the official website

Installation

Now that we have Redis running in our system, we can now grab the node package:

npm i redis

It's time to test it out!
We can write a new command that will set a temporary key with the data for us.

NOTE: Redis values can store strings, hashes, lists, and so on. We're just interested in strings so for this we'll have to stringify our data before storing it.

Usage

We could write two simple functions that will take care of writing and reading data from Redis:

const { promisify } = require('util');
const redis = require('redis');
const client = redis.createClient();

/* Promisfy so we can have promise base functionality */
const getAsync = promisify(client.get).bind(client);
const setAsync = promisify(client.set).bind(client);
const setexAsync = promisify(client.setex).bind(client);
const ttlAsync = promisify(client.ttl).bind(client);

client.on('error', function (error) {
  console.error(error);
});

/**
 * Writes strigify data to cache
 * @param {string} key key for the cache entry
 * @param {*} value any object/string/number */
const cacheSet = async (key, value) => {
  return await setAsync(key, JSON.stringify(value));
};

/** Retrieves data for a given key
 * @param {string} key key of the cached entry */
const cacheGet = async (key) => {
  const data = await getAsync(key);

  return JSON.parse(data);
};

module.exports = async (msg, key, value) => {
  await cacheSet(key, value);

  return msg.channel.send(`We just stored: key: **${key}** | value: **${value}**`);
};

Now we can tell Redis to store some data under a specific key.

NOTE: this is not just with discord.js, you can apply the same thing to any Node.js APP or even your own API

Let's try it out by storing a new value from our command, a name for example:

We can check our Redis instance to be sure that we actually store it. We'll use the built-in redis-cli:

Run redis-cli and we'll get something like this:

  /data> redis-cli
  127.0.0.1:6379>

Use KEYS * to recive all our stored keys

  127.0.0.1:6379> KEYS *
  1) "username"
  127.0.0.1:6379>

Use GET username to retrieve our stored value

  127.0.0.1:6379> GET username
  "\"en3sis\""
  127.0.0.1:6379>

This is what we expected to happen. Now we can get to the fun part and unlock all the potential.

Cache data from APIs

For this demo, we'll use a free Weather API. Later in the article, we'll explore some real-world examples where this approach shines.

Preparation

We'll install Axios HTTP client to fetch the API (you can use whatever else. npm install axios ) and create a function that will allow us to fetch the API.

/**
 * Fetch from the Weather API endpoint
 * @param {string} city - City to be fetched
 */
const fetchData = async (city) => {
  const { data } = await axios.get(`https://goweather.herokuapp.com/weather/${city}`);

  return data;
};

We'll change our command to grab the data from the API and send some of the stats to the chat.

// New code addition
const axios = require('axios');

// ...

// New code addition
module.exports = async (msg, key) => {
  const currentWeather = await fetchData(key);

  return msg.channel.send({
    embed: {
      title: `Weather in ${key}`,
      fields: [
        {
          name: ' 🌡 Temp:',
          value: `**${currentWeather.temperature}**`,
          inline: true,
        },
        {
          name: '🍃  Wind:',
          value: `**${currentWeather.wind}**`,
          inline: true,
        },
      ],
      color: 0x03a9f4,
    },
  });
};

If we run the command we'll get the following result:

Sometimes I miss going back to Spain, you can tell why based on the weather :P

Solving the problem

Let's imagine that our API has a rate limit of 1000 requests per month. With the current implementation, we could only serve 1k requests and not a single extra one.

Now imagine our Bot is part of multiple Guilds and multiple users are using our new command. If user 1 fetches the data for Almería, a beautiful city located in the southeast of Spain on the Mediterranean Sea, we could store this data for 1h for example. We don't really need fresh data (every 10 min, and few paid APIs allow you to do so).

Now, when user 2 in another server also wants to see the weather in Almería, we'll fetch the data from our local, in memory and blazing fast (~1ms response time) Redis cache.
For the following hour we could show the weather in Almería for 1 billion users and we only spent one single HTPP request!

Implementation

You notice that I mentioned the persistence of the data, another great build-in function that Redis has is TTL (time to live) where you can specify for how long you want some data to be cached, without having to worry about cronjobs, re-validation, and so on.
We'll add a new function that will cache some data for the amount of time we indicated:

/**
 * Writes strigify data to cache
 * @param {string} key key for the cache entry
 * @param {*} value any object/string/number
 * @param {number} ttl cache duration in seconds, default 3600 (1h) */
const cacheSetTTL = async (key, value, ttl = 3600) => {
  return await setexAsync(key, ttl, JSON.stringify(value));
};

Now we can refactor our code so that every time we want to retrieve the weather from a given City, we first check the cache. If the city is in the cache, we use that data. If it's not in the cache, we'll fetch the data from the API and save the copy in our Redis instance. We can implement this directly in our fetchData() function.

/**
 * Fetch for the Weather API endpoint
 * @param {string} city - City to be fetched
 */
const fetchData = async (city) => {
  const isCached = await cacheGet(city);

  if (isCached) {
    console.log('⚡️  From cache');

    return isCached;
  } else {
    // Fetch data
    const { data } = await axios.get(`https://goweather.herokuapp.com/weather/${city}`);

    // Save data to cache
    await cacheSetTTL(city, data);

    return data;
  }
};

The whole code can be found here

And we're done! We can now run our command, check for the weather in a given city and return the already cached data or fetch and store it.

When we run our command, it will:

Check for the KEY in Redis
It won't find it so it will make the HTTP request to the API
Saves the data in Redis using the city as KEY
Return the data from our fetchData() function and send the embed

For the second time we (or another user) use the command, it will grab the data directly from the cache.

# In Discord
> cache nuremberg
> cache nuremberg
> cache nuremberg

# We should see in our application a console log saying:
Logged in as Hans!
⚡️  From cache
⚡️  From cache

For the first command, we fetch and store the data, for the following commands we serve the data from the cache.

NOTE: You can always check the remaining time left by using the redis-cli or the ttlAsync function:

127.0.0.1:6379> KEYS *
1) "nuremberg"
127.0.0.1:6379> TTL nuremberg
(integer) 3370 # remining time in seconds
127.0.0.1:6379>

Wrapping up

I hope that this walkthrough helped you get a better understanding and given you some ideas on how to handle the sometimes annoying rate limits.

Real-world usecases

As promised before, here some examples of when this is really useful.

When dealing as in our example with APIs like the weather where we want to reuse the most amount of data with spending a single request.
Games APIs: I used it to fetch data from games like Battlefield and reuse the data for things like players' comparison. If I want to see user A stats, then user B used the command to see him and I decide to compare our profiles, see how's doing better I can run the command with something like !bf userA userB and instead of doing two requests to the API to get each player stats, I used the data already available in my cache.
Same as before, one of the commands is the COVID-19 stats. I also cache the data for a given country (since it's updated once per day) so I can reuse the cache data when another user from a different server fetches the data from the same country.
Dashboard and Discord API: Discord only allows you to fetch the API by sending an `x ammount of requests per second. While working with a Dashboard where you need to fetch the Guild channels, users, roles... you don't want to do it every time you load a Guild dashboard. For it, I only do it once and set a TTL of ~2mins for some parameters.

Finally

As always, you can find the code with all examples at https://github.com/en3sis/discord-guides

Any feedback, questions, or suggestions are welcome!
Thanks for reading! ~ https://twitter.com/en3sis

Advanced Discord.js: Custom embeds using attachments

Sorin Curescu — Sun, 17 Jan 2021 22:07:24 +0000

Using SVGs
Using HTML & CSS
Conclusion

Looking for a Discord.JS bot? Check out Hans! Now it's open-sourced 🥳!

While dealing with sending messages with our Discord Bot, we can do it in multiple ways such as text, attachments and the most common way, embeds.

One example could be something like the following command that will display the weather based on a location:

Sometimes using the MessageEmbed class it is enough, but we may want to represent the data in a different layout/design:

I was seeing several others bots making use of custom ways to show things like stats, users profiles, etc., but I wasn't sure how that can be done and I couldn't find any examples. Some ideas came to my mind and it was time to try things out :P

The following examples are built on top of Discord.js, a Node.JS wrapper to interact with the Discord API. For more information related to Discord.js and how you can start building your own BOT, you can follow along with this amazing series created by Daniel Shiffman.

Using SVG

When I first started looking into this, it came to my mind that having an SVG-to-PNG library could work pretty well since we can have the SVG in a template string and replace the placeholders with our data (user input, API calls, etc...) and then make use of MessageAttachment to attach the output.

For the following SVG image,

I was able to find a library that would take an SVG as a string value and return a buffer


 install svg-png-converter

and then we could have a function to handle all the functionality, in this case we'll have it in its own file and we call it svgToPng.js (Discord only allows attachments of images as JPEG/PNG formats):



// svgToPng.js
const { MessageAttachment } = require('discord.js')
const { svg2png } = require('svg-png-converter')


module.exports = async (msg, name) => {

  const outputBuffer = await svg2png({
    input: `<svg xmlns="http://www.w3.org/2000/svg" width="350" height="136" viewBox="0 0 350 136">
  <g id="template" transform="translate(-208 -209)">
    <rect id="background" width="350" height="136" transform="translate(208 209)" fill="#232323"/>
    <text id="_usr_" data-name="${name}" transform="translate(326 286)" fill="#fff" font-size="20" font-family="SegoeUI, Segoe UI"><tspan x="0" y="0">Hello ${name}</tspan></text>
    <path id="icon" d="M7.5-16.68,15-13.32v5a10.351,10.351,0,0,1-2.148,6.348A9.33,9.33,0,0,1,7.5,1.68,9.33,9.33,0,0,1,2.148-1.973,10.351,10.351,0,0,1,0-8.32v-5Zm1.758,4A2.435,2.435,0,0,0,7.5-13.4a2.435,2.435,0,0,0-1.758.723A2.361,2.361,0,0,0,5-10.918a2.425,2.425,0,0,0,.742,1.777A2.4,2.4,0,0,0,7.5-8.4a2.4,2.4,0,0,0,1.758-.742A2.425,2.425,0,0,0,10-10.918,2.361,2.361,0,0,0,9.258-12.676ZM7.5-6.836a8.754,8.754,0,0,0-2.031.273,6.19,6.19,0,0,0-2.051.9A1.74,1.74,0,0,0,2.5-4.258,6.007,6.007,0,0,0,4.707-2.383,5.947,5.947,0,0,0,7.5-1.6a5.947,5.947,0,0,0,2.793-.781A6.007,6.007,0,0,0,12.5-4.258a1.486,1.486,0,0,0-.547-1.094,4.2,4.2,0,0,0-1.348-.82A10.513,10.513,0,0,0,8.984-6.66,7.147,7.147,0,0,0,7.5-6.836Z" transform="translate(302 286)" fill="#fff"/>
  </g>
</svg>
`,
    encoding: 'buffer',
    format: 'png',
    quality: 1
  })
// for more configuration options refer to the library

  return msg.channel.send(`This is a test:`, new MessageAttachment(outputBuffer, '${name}.png'))

 }
}

The result:

This was working great with small illustrations where there are just some small changes in the text. For other things, it was not good enough (when you want to dynamically show an image based on some value, such as a list of different icons for each different user role). While this still could be achieved with the SVG, it felt that it's too much work and it wasn't the right fit.

It was time to go back to the drawing board and to think about some other ways to achieve this.

Using HTML & CSS

In the same way, we deal with SVG to PNG conversion, it would be possible to render a webpage with all our data, assets and take a screenshot of it (using puppeteer or some other libraries) and finally attach the screenshot? This would solve many issues and creating a layout in the front-end it's something that I understand and I feel comfortable with, so it sounded like a great idea!

It ends up that you can! After a quick search in the infinitely large npm registry and I was able to find a library that does exactly that (node-html-to-image).

Now the only thing we have to do is to build the layout for our example (HTML & CSS) and to put everything together. We'll make use of the avatar API to dynamically generate an avatar based on the user's name input.

Let start building a quick mockup for our card:

Now that we have our HTML & CSS, we can start working on generating an image from it using the NPM package:


 install node-html-to-image

and in our htmlToPng.js file we have:



//  htmlToPng.js

const { MessageAttachment } = require('discord.js')

const nodeHtmlToImage = require('node-html-to-image')

module.exports = async (msg, name) => {

const _htmlTemplate = `<!DOCTYPE html>

<html lang="en">

  <head>

    <meta charset="UTF-8" />

    <meta name="viewport" content="width=device-width, initial-scale=1.0" />

    <meta http-equiv="X-UA-Compatible" content="ie=edge" />

    <style>

      body {

        font-family: "Poppins", Arial, Helvetica, sans-serif;

        background: rgb(22, 22, 22);

        color: #fff;

        max-width: 300px;

      }

  .app {
    max-width: 300px;
    padding: 20px;
    display: flex;
    flex-direction: row;
    border-top: 3px solid rgb(16, 180, 209);
    background: rgb(31, 31, 31);
    align-items: center;
  }

  img {
    width: 50px;
    height: 50px;
    margin-right: 20px;
    border-radius: 50%;
    border: 1px solid #fff;
    padding: 5px;
  }
&lt;/style&gt;



</head>

  <body>

    <div class="app">

      <img src="https://avatars.dicebear.com/4.5/api/avataaars/${name}.svg" />

  &lt;h4&gt;Welcome </span><span class="p">${</span><span class="nx">name</span><span class="p">}</span><span class="s2">&lt;/h4&gt;
&lt;/div&gt;



</body>

</html>

`

const images = await nodeHtmlToImage({

    html: _htmlTemplate,

    quality: 100,

    type: 'jpeg',

    puppeteerArgs: {

      args: ['--no-sandbox'],

    },

    encoding: 'buffer',

  })

// for more configuration options refer to the library

return msg.channel

    .send(new MessageAttachment(images, </span><span class="p">${</span><span class="nx">name</span><span class="p">}</span><span class="s2">.jpeg))

}

The result:

Conclusion

Now we have two new ways to generate more complex and good looking cards for your Discord users to consume. One good example of this would be to display some game stats for example. My brother and his friends play World of Warcraft, so I took a little time to create a command, that will display the main stats for their characters using the HTML & CSS technique, here is the result of it:

I would suggest using the SVG method while dealing with a static design/elements but only some text changes and use the HTML one while dealing with dynamic lists or images.

In the previous example, we might be able to archive the same result using an SVG but I find it hard since each element (the item icon and his value) are coming from an API in a different format (JPG/PNG for the images). Appending the item to a list it's way easier than applying a block of code to the SVG bases on some condition. Also styling things like images wrappers (round images with round borders) it's way easier in HTML & CSS than with SVG using things like masks.

I hope that this guide will open new ways for your creativity, which I would love to see them.

You can find the BOT code with all examples at https://github.com/en3sis/discord-guides

Any feedback, questions or suggestions are welcome!
Thanks for reading! ~ https://twitter.com/en3sis

DEV Community: Sorin Curescu

Advanced Discord.js: Cache APIs Requests with Redis

Table Of Contents

Getting started

Installation

Usage

Cache data from APIs

Preparation

Solving the problem

Implementation

Wrapping up

Real-world usecases

Finally

Advanced Discord.js: Custom embeds using attachments

Table Of Contents

Using SVG

The result:

Using HTML & CSS

The result:

Conclusion