DEV Community: Santhosh Reddy

Performance benchmarks - redis mget vs pipeline

Santhosh Reddy — Mon, 03 Oct 2022 14:45:56 +0000

This post was originally published at distributedstack.

Redis provides a benchmarking tool called redis-benchmark for testing the performance of common redis commands. The benchmarks in this blog are run on below configuration.

Model - MacBook Air 2019
Processor - 1.6 GHz Dual-Core Intel Core i5
Memory - 8 GB 2133 MHz LPDDR3
Client - redis-benchmark
Server - localhost:6379

Both client and server are running on the same machine
redis-benchmark runs the tests over 100000 commands by default. You can tweak the numbers as required.

If you haven’t already, do checkout this article for more details on redis-benchmark tool.

Below benchmark runs 1000 commands in each pipeline over all the available tests.

$redis-benchmark -q -P 1000

PING_INLINE: 200803.22 requests per second
PING_BULK: 401606.44 requests per second
SET: 236406.61 requests per second
GET: 277008.31 requests per second
INCR: 229885.06 requests per second
LPUSH: 184501.84 requests per second
RPUSH: 220750.55 requests per second
LPOP: 183150.19 requests per second
RPOP: 223713.64 requests per second
SADD: 224719.11 requests per second
HSET: 172117.05 requests per second
SPOP: 335570.47 requests per second
LPUSH (needed to benchmark LRANGE): 186219.73 requests per second
LRANGE_100 (first 100 elements): 18278.19 requests per second
LRANGE_300 (first 300 elements): 6242.20 requests per second
LRANGE_500 (first 450 elements): 4006.73 requests per second
LRANGE_600 (first 600 elements): 2906.55 requests per second
MSET (10 keys): 29299.74 requests per second

So, we could do ~277k reqs/sec with redis get command in pipeline.

Let’s set some keys to benchmark redis mget

set hash1_key1 "The golden rule of a useful benchmark is not to compare apples to oranges"
set hash2_key2 "The golden rule of a useful benchmark is not to compare apples to oranges"
set hash3_key3 "The golden rule of a useful benchmark is not to compare apples to oranges"
$redis-benchmark mget hash1_key1 hash2_key2 hash3_key3 ... (1000 keys)

  100000 requests completed in 113.54 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

0.00% <= 1 milliseconds
0.06% <= 2 milliseconds
15.38% <= 44 milliseconds
20.01% <= 45 milliseconds
25.82% <= 46 milliseconds
32.60% <= 47 milliseconds
40.24% <= 48 milliseconds
48.40% <= 49 milliseconds
56.21% <= 50 milliseconds
63.03% <= 51 milliseconds
68.34% <= 52 milliseconds
72.26% <= 53 milliseconds
75.10% <= 54 milliseconds
90.14% <= 78 milliseconds
95.77% <= 83 milliseconds
99.99% <= 234 milliseconds
100.00% <= 235 milliseconds
100.00% <= 236 milliseconds
100.00% <= 237 milliseconds
100.00% <= 241 milliseconds
880.72 requests per second

So, we were able to fetch 880*1000 = 880k keys/sec via mget compared to 277k keys/sec with get in pipeline.

Working with unpublished node dependencies

Santhosh Reddy — Mon, 27 Jul 2020 16:19:45 +0000

If you are a Node.js developer, you might've ended up in a situation where you want to use an unfinished feature from another node dependency.

Let's elaborate this a bit. For example, your entire project is broken logically into 4 npm modules. One module, which is the main one depends on the other 3 modules. With this setup, let's get changes from sub-modules to integrate with the main module.

The simplest way is to publish the sub-modules to npm. Use the new versions in your main node module's package.json file and re-install. Well, the downside with this approach is if you have made a mistake in your sub-modules, you have to re-publish and use them accordingly. But, things don't stop there. You will have to repeat this until your main node module is stable. Headache. Right? I know.
So, How do we get around this problem?

Using npm link

With this approach, you can work with any node dependencies if they are checked out at some location in your local machine. All you have to do is run below command in the root folder of the package, which is a dependency for your main node module.

npm link

Alright! What does this do? If you have worked on node.js projects, you know there is a node_modules folder that has your installed dependencies. Similarly, there is a global folder for the dependencies. The above command creates a symbolic link for the package in which this command is run in the global folder. You have to run this command again in the package where you want to use the dependency code with the name in package.json.

npm link <package-name>

With this, any changes you make to your dependency node module will be used directly without having to re-install. The above 2 steps can be made short with just below command.

npm link <relative-path-to-the-dependency>

Getting the source from GitHub

Now, let's discuss another use case where you are not the one working on your dependency, but a colleague of yours. And he doesn’t want to publish the code until he makes sure the feature is complete to some extent. But you need this person's code to test any early-stage integration issues. I am assuming you both use the git version control system for managing your code. You can get the changes your colleague has pushed to git with the link to the repository code as below in your package.json file.

'package-name': 'git@github.com:<repository-name>.git#<branch-name>'

Once you've placed the above path in package.json file, you need to run a clean npm install to get the latest code from git.

Comment down if you using any other approach?

Is Node.js really single-threaded?

Santhosh Reddy — Sun, 26 Jul 2020 07:22:49 +0000

Many developers wonder at some point in their Node.js programming journey that, is Node.js really single-threaded? If it is, how is it able to do asynchronous execution? Some people understand it in terms of even loop, but there is more to it. I hope you understand what event loop is since you are reading this. But, if you don’t, check out this awesome talk by Philip Roberts.

In Chrome, Javascript runs in a single thread. You are sharing the same thread for rendering as well as your code execution. Chrome actually employs multi-process architecture, which means that each tab, plugin in your browser is getting a new process. And any crash or block is not effecting others. When JS code is executed in the browser, it is essentially blocking the browser painting or render since there is no thread to do the same. But before it picks something from the callback queue after the current call stack is empty, browser paint actually takes precedence. That’s about the browser. But, how do things go in a backend JS runtime, Node.js?

The answer to the above question, Node.js is single-threaded similar to Javascript. So, your JS code is running in a single thread(main thread) in consensus with the event loop and callback queue. But, Node.js is not purely Javascript code. Some of it is C++ code, some things when done asynchronously like network calls, file system tasks, DNS lookup, etc. are actually not handled by the main thread. Node.js does this optimization of using C++ code(which has access to multiple threads) internally for these tasks, which helps in reducing the execution time when used correctly. Node.js actually does offload the asynchronous tasks to C++ code, where it has the provision to use multiple threads to speed up execution, but if you force it to be sync(using only the synchronous version of the API), you are binding it not to do this optimization.

Backend developer roadmap, skills, resources

Santhosh Reddy — Sat, 25 Jul 2020 08:49:53 +0000

Backend developers mostly deal with data, business logic, and the efficient retrieval of data from variety of sources. The list doesn't end here. There are a number of things that a good backend developer is expected to be adept at. This roadmap tried to cover some of them. This post will try to list down the resources instead.

Internet and basic frontend knowledge

Mozilla Developer Network(MDN) docs is a great start, if you are trying to start taking the web development path. I've given some important links below but you can explore the entirety, there is a ton of information that is up to the point and well explained.

How the web works
HTTP protocol
Speaking the browser language(HTML-CSS-Javascript)
Javascript, everything

Although the below resources are not for backend developers, some of you might be interested in some frontend frameworks that guys in your company are using.

Bootstrap
React documentation

OS and basic machine usage skills

How OS works in general
Threads and concurrency
List of linux commands
Inter-process communication

Programming languages and frameworks

Although it is not necessary that you learn every language in the world, it is good to have skills in at least one compiled, scripting language and choose one for a paradigm. Languages are essential tools for programmers and they deserve a section in backend developer roadmap and resources list.

Ruby
Java
Python
Javascript

Ruby on Rails
Node.js
Django
React - optional

Version control

You won't write clean code on your first iteration most of the time. Also, if you happen to be in a situation where you lost your code due to variety of reasons, then version control systems got you covered.
Git and Mercurial are among the most popular version control systems out there. You should definitely learn Git. Below are some resources where you can start.

Git basics
Git documentation

Databases(relational) and ORM

As we discussed already, backend developers mostly deal with data. Hence, it is essential that you learn several ways to work with the same. Relational databases were the goto for every use case. But, that has changed over the years, and several purpose built databases took its place. Initially you should be able to interact with the relational databases using SQL, or an ORM. But, as you get more comfortable, you should explore the database internals like how indexes work, how joins will effect performance, transactions, isolation levels, and designing for availability and consistency etc.

SQL tutorial
n+1 select
MySQL architecture and History
Aurora Deep Dive
Rick Houlihan's talk on matching the workload to DB
Rails ActiveRecord - ORM
Django Queryset - ORM

Databases(NoSQL)

NoSQL databases came into picture to solve several problems of relational databases like schema rigidity, extensibility, scalability etc. There are several databases in this space. I will list down some of them and resources to learn about them.

MongoDB from it's CTO
Advanced design patterns for DynamoDB(or any NoSQL in general)
Single table design - DynamoDB
Why Dynamo scales where relational databases don't
Different take on NoSQL vs SQL

Databases(Graph)

There is another class of purpose built databases which puts relations in the data, forefront. These databases exploit the related nature of the data and help answer complex queries easily. These databases are the goto when you want to answer queries like friends of friends, what do my friends like, efficiently. Below are some resources where you can learn about these.

Graph databases for beginners series
Amazon Neptune
Introduction to cypher
Cypher reference card

Big data(optional)

Everything you need to know about Big Data
Analytics at Netflix
Netflix Metacat

APIs and authentication

REST cookbook
REST wiki
Roy fielding's
RPC vs REST
REST over RPC
gRPC talks

More and more companies are finding value in GraphQL recently. You can learn about the same below.
GraphQL
GraphQL vs REST

Web security fundamentals

Web application security should not be an after thought. Many developers focus much on the application logic, since the code is generally audited by a security researcher for an initial release. But, developers should write code keeping security in perspective always. The first link below is more relevant to the Rails ecosystem, but it will cover some general security best practices. Be sure to check it out.

Securing Rails applications
SSL and TLS
Discussion on SSL and TLS
Virtual private clouds explained
Cross site request forgery
Django, CSRF
OWASP attacks list
Cross Origin Resource Sharing(CORS)
Content security policy
HTTP Basic Auth
Cookies and security
DOS attack

If you are working with node.js, you can take advantage of npm audit to find out and fix known vulnerabilities in npm modules.

If you like the blog, follow me at @distributedmonk for more backend tech content and do checkout the blog.