DEV Community: Shahid Shaikh

Redistill - High Performance Redis Compatible Key Value Database

Shahid Shaikh — Wed, 04 Feb 2026 08:36:23 +0000

Redis has always been one of my go-to KV databases. I strongly believe in — and advocate for — a memory-first, disk-later approach when designing high-throughput, low-latency systems. Redis is fast and durable, but I kept wondering: can we build something Redis-like that’s even faster?

That curiosity pushed me to experiment deeply with Redis and the RESP protocol. Eventually, I decided to “distil” Redis — strip it down to the essentials and build a lighter version that supports only the commands I use most often (Strings and Hashes) — and see if we could push throughput and latency even further.

Turns out, we did.

I’m happy to announce that I’m working on an open-source project called Redistill — a distilled Redis. It’s written in Rust and based on benchmarks run on AWS c7i.16xlarge (Intel, 64 cores, 128GB RAM) using memtier_benchmark, I can confidently say this is the fastest single-instance, Redis-compatible database available today.

Project link: https://github.com/redistill-io/redistill

This is just the beginning. The project is barely 8 weeks old. I still need to figure out clustering and other production-ready features, but so far, as a single instance, this is ready to use ina production environment.

Motivation

The entire motivation behind the project was to build my own KV database that works faster than Redis and supports the commands I use frequently. It doesn't need to be versatile like Redis, but it does one thing best - the fastest KV database possible.

I also wanted to learn Rust, and this was a good opportunity to revisit the fundamentals of system design and build a database from scratch (not entirely scratch, I use RESP protocol).

Key Characteristics

Redistill supports the RESP protocol, so there is no need for code changes on the client side. Your existing code, which works with Redis, will work with Redistill, assuming you use commands Redistill supports.

Below are some key features:

High-performance in-memory key-value database
Redis protocol compatible (RESP) - works with all Redis clients
9.07M operations/second - 4.5x faster than Redis, 1.7x faster than Dragonfly
5x lower latency (p50: 0.48ms vs Redis 2.38ms)
Multi-threaded architecture with lock-free reads
Optional persistence - Snapshot support for warm restarts (disabled by default for max performance)
Production-ready security and monitoring features

Why not all Redis commands?

This was intentional.

I wanted a KV database for my personal and work uses. I used String and Hash commands almost 90% of the time. So keeping it lightweight was the design and not a restriction.

Will I add more commands in the future? Of course, this is a FOSS project, so I'll let contributors add the commands as they seem fit.

Why Redistill?

High-Performance KV Database - Fastest in-memory key-value database available
Maximum Single-Instance Performance - 4.5x faster than Redis, 1.7x faster than Dragonfly per instance
Lower Latency - Sub-millisecond p50 latency (0.48ms)
Cost Efficient - 50-83% infrastructure savings (fewer instances needed)
Drop-in Compatible - Works with existing Redis clients
Production Ready - TLS, authentication, monitoring, health checks
Multi-threaded - Utilises all CPU cores efficiently
Optional Persistence - Snapshot support for warm restarts when needed (disabled by default)

Quick Start

We recommend using Docker to quickly spin up Redistill in your machine or server.

# Run with default settings
docker run -d --name redistill -p 6379:6379 shahidontech/redistill:latest

# Test it works
redis-cli ping
# PONG

redis-cli set hello world
# OK

redis-cli get hello
# "world"

We also have other methods, such as Homebrew and building from source.

Performance

Competitive Benchmark (c7i.16xlarge)
Independent comparison on AWS c7i.16xlarge (Intel, 64 cores, 128GB RAM) using memtier_benchmark with production-like configuration:

Test Configuration:

Duration: 60 seconds
Threads: 8, Connections: 160 (20 per thread)
Pipeline: 30, Data size: 256 bytes
Workload: 1:1 SET: GET ratio

Metric	Redistill	Dragonfly	Redis	vs Redis	vs Dragonfly
Throughput	9.07M ops/s	5.43M ops/s	2.03M ops/s	4.5x	1.7x
Bandwidth	1.58 GB/s	923 MB/s	337 MB/s	4.7x	1.7x
Avg Latency	0.524 ms	0.877 ms	2.000 ms	3.8x faster	1.7x faster
p50 Latency	0.479 ms	0.807 ms	2.383 ms	5.0x faster	1.7x faster
p99 Latency	1.215 ms	1.975 ms	2.959 ms	2.4x faster	1.2x faster
p99.9 Latency	1.591 ms	2.559 ms	4.159 ms	2.6x faster	1.6x faster

Key Observations:

Redistill processed 544M total operations (2.7x more than Dragonfly, 4.5x more than Redis)
Consistent low latency across all percentiles
No errors or connection issues across all systems

Use cases

Redistill is a high-performance key-value database perfect for applications requiring maximum speed and minimal latency.

Perfect For
Session Storage

1M+ operations/second
Sub-millisecond latency
Automatic TTL expiration
60% cost reduction vs alternatives

API Response Caching

95%+ cache hit rates
50-150x faster than database queries
Automatic memory management
LRU eviction is built in

Rate Limiting

Millions of counters
TTL-based cleanup
High write throughput
Perfect for API gateways

Real-time Leaderboards

Fast reads for rankings
Periodic score updates
Can rebuild from the database
Sub-millisecond queries

Not Recommended For

Critical persistent data (optional snapshots available, but not real-time durability)
Financial or transactional data (use ACID-compliant database)
Data that cannot be regenerated (use a database)

Examples

The Redistill uses the RESP protocol to communicate, so any Redis client library should be able to connect to Redistill and communicate as they are communicating with Redis with the same commands.

For example, in Python:

import redis
r = redis.Redis(host='localhost', port=6379, password='your-password')
r.set('key', 'value')
value = r.get('key')

In Node.js:

const Redis = require('ioredis');
const redis = new Redis({host: 'localhost', port: 6379, password: 'your-password'});
await redis.set('key', 'value');

Contributing

Contributions are welcome! Please:

Open an issue to discuss proposed changes
Follow Rust coding conventions
Include tests for new features
Update relevant documentation

Project link: https://github.com/redistill-io/redistill

10 ChatGPT Prompts to Boost Developer Productivity

Shahid Shaikh — Fri, 10 May 2024 16:38:43 +0000

As developers and engineers, we constantly seek ways to streamline our workflows, increase productivity, and solve complex problems efficiently. With the advent of advanced language models like ChatGPT, we now have powerful tools to assist us in our daily tasks.

By leveraging the capabilities of ChatGPT, we can generate prompts that enhance our productivity and creativity, making us more effective problem solvers and innovators.

In this article, we’ll explore 10 ChatGPT prompts tailored specifically for developers and engineers to boost their productivity and streamline their workflow.

Code Refactoring Suggestions

Here is the sample prompt:

*“I have a code that needs refactoring. Can you provide suggestions to improve its readability and efficiency? Here is the code: ”
*

Use ChatGPT to generate recommendations for refactoring code snippets, such as identifying redundant lines, suggesting better variable names, or proposing alternative algorithms to optimize performance.

Please refer to the screenshot below:

Suggest better code using ChatGPT prompts

Here is the response:

ChatGPT response

Troubleshooting Assistance:

Here is the sample prompt:

*“I’m encountering an error message [insert error message here] in my code. Can you help me troubleshoot and find a solution?”
*
This prompt will help you troubleshoot bugs or issues in your code. Again, there may need a couple of iterations to really nail down the problems but this is a good starting prompt.

API Documentation Retrieval

Here is the prompt:

*“I’m working with the [insert API name] API. Can you provide me with relevant documentation or usage examples?”
*
This is really helpful when we are working with new systems or platforms and instead of reading all the documentation, you can ask the ChatGPT to retrieve useful information for you in a summarized way.

Design Pattern Recommendations

Here is the prompt:

*“I’m designing a new software component. Here is the requirement: [ Put your requirement here]. What design pattern would you recommend for implementing [insert functionality]?”
*
This prompt requires a good level of detail but it can help you recommend some of the best design patterns you should use for your problem set.

Algorithm Optimization Techniques

Here is the prompt:

*“I’m implementing [insert algorithm name]. Are there any optimization techniques or best practices I should consider?”
*
This is not only limited to algorithms but you can use some code as well. In short, this prompt will help you optimize the algorithm/code.

Code Review Feedback

Here is the prompt:

*“I’ve written a new feature. Can you review my code and provide feedback on potential improvements? Here is the code : [Insert code here]”
*
ChatGPT can provide some really good feedback about your code. You may or may not implement all those feedbacks but it can certainly be a good starting point.

Library or Framework Recommendations

Here is the prompt:

*“I’m starting a new project. Can you recommend a suitable [insert programming language] library or framework for [insert functionality]?”
*
ChatGPT can suggest popular libraries, frameworks, and tools based on the programming language and desired functionality, enabling you to make informed technology choices.

Technical Documentation Summaries

Here is the prompt:

*“I need a summary of the [insert technology or concept] technical documentation. Can you provide a concise overview?”
*
This is my most used prompt, I summarize the technical documentation and read the gist, this has certainly improved my productivity.

Code Snippet Generation

Here is the prompt:

*“I need a code snippet for [insert functionality or task]. Can you generate a sample code snippet?”
*
This is a good prompt to generate a starter code pack. But be sure not just to copy the code and use it, be cautious with the code generated by LLMs as they contain security flaws and bugs as well.

Project Planning and Task Prioritization

Here is the prompt:

*“I’m planning my project roadmap. Can you suggest a prioritized list of tasks based on [insert project requirements or constraints]?”
*
ChatGPT can analyze project requirements, dependencies, and deadlines to generate a prioritized task list, helping you effectively manage project timelines and deliverables.

Conclusion:
Incorporating ChatGPT prompts into your development workflow can significantly enhance productivity, creativity, and problem-solving capabilities. By leveraging ChatGPT’s natural language understanding and generation capabilities, developers and engineers can streamline tasks such as code refactoring, troubleshooting, documentation retrieval, and project planning. By integrating ChatGPT into your toolkit, you empower yourself to tackle challenges more effectively and unlock new levels of innovation in your projects.

This article originally posted at https://shaikhshahid.com/blog/10-chatgpt-prompts-to-boost-developer-productivity/79/

Exploring Codium GitHub Pull Request AI Assistant

Shahid Shaikh — Wed, 27 Dec 2023 08:58:25 +0000

I work in a software development and consulting firm and a large portion of my day is spent reviewing pull requests from different projects. Pull requests a.k.a merge requests are the contributions done by the engineers, generally covered by the unit tests.

My job is to ensure the pull requests follow our standards and do the review of the code or delegate to one of the architects to review it if my plate is full or I am busy with some other work.

Here are some of the standards that I make sure everyone is following:

The Jira ticket link is mentioned in the PR title.
The ticket description is not vague or one-liners and explains the changes and how it is tested.
Unit test coverage (more than 80%).

And some more project-related details. Every developer in my team always cries over filling in all these details and they always say “Look at the code”. I understand this frustration as I used to bash the same to my seniors.

I use Github copilot and some of my team members also use the same. It’s a fantastic tool that provides real-time code suggestions and helps us write code/tests faster.

GitHub announced copilot for PR and we were excited about it. While this is on the waiting list and should be available soon, I got the article suggestion on dev.to platform about a similar tool, opened the article and was astonished to know that the team at Codium has already done this, and they did it better!

After reading and using the Codium PR description for a day, I wanted to document and write a blog about it for fellow developers and engineers.

Making Pull Requests Less Painful Using Codium PR AI Agent

Codium AI and Github Copilot both use artificial intelligence to assist developers in writing better pull request descriptions and titles, detecting security flaws and much more.

One of the key differences between them is that Codium provides open-source solutions that can be integrated into our infrastructure. This is a major advantage over Github because the cost of a subscription to Copilot is high when you have a team of say 20 developers.

There are other key differences as well and they are the key factors why you should consider Codium like I did over GitHub PR tool.

One major difference is that Codium supports all Git-based platforms. I have many clients who use self-hosted Gitlab and they just simply cannot use Copilot because it’s supported only on GitHub. This was one of the key reasons why we started researching such tools.

The open-source version of the Codium PR agent is one of the best approaches to using Codium in the infrastructure maintained by the team. We used the docker image to install this and it was easy as cake! There are other ways to install as well so you are not limited by options.

How PR Agent Works

Here is the detailed diagram from the official Codium website.

Once the PR agent is installed in Github or any supported Git-based software. You can invoke the PR agent by using different commands such as:

/review - Reviews the PR and provides you with gist information.
/describe - generates a description of the PR.
/ask ? - ask any question such as “suggest any fixes”

And more.

The GitHub PR tool will provide a similar experience and we will know more about it once it’s live for everyone. I would rather recommend using Codium because of its seamless integration within a system and provides support to almost all programming languages.

Using PR Agent in Github

Let’s try to use Codium’s PR agent in GitHub and see how it adds value to our workflow. I am going to use a public repo in my profile to show you how it works. For the demo, we are going to use the pre-described Codium PR Agent image in the GitHub actions.

You can refer to the documentation here as you follow the article.

First thing, create a .github/workflow folder in the code repository. Inside the workflow folder, create a file named pr_agent.yml and place this code.

  pull_request:
  issue_comment:
jobs:
  pr_agent_job:
    runs-on: ubuntu-latest
    permissions:
      issues: write
      pull-requests: write
      contents: write
    name: Run pr agent on every pull request, respond to user comments
    steps:
      - name: PR Agent action step
        id: pragent
        uses: Codium-ai/pr-agent@main
        env:
          OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

You need an openAI API key, you can obtain it by signing up here and creating an API key.

Now, take the API key and open the Settings page of your GitHub repository. Go to Secrets and Variables -> Actions. Here create a new repository access add the key name OPENAI_KEY and place your openAI API key.

There you go, now we can use the Codium PR agent. The Github token will be created automatically by Github so we don’t need to worry about it.

Now go ahead and create a pull request in your repository. Once your pull request is created, the PR agent will automatically add a comment mentioning a bit of detail. Here is the screenshot for reference.

Now, let’s see how the PR agent can provide us with more details. To add PR agent commands, we need to add a command with the commands in it.

I am adding the first command as /describe and the PR agent will generate and add a comment to the pull request as an output.

Here is the output.

Awesome!

Now, let’s generate labels for this pull request. Add /generate_labels to the comment box and let the PR agent do the magic.

PR agent automatically analyzed and added the label Tests to the pull request.

Let’s try another command and try to generate the changelog. Add /update_changelog and you should have a comment added as an output.

Go ahead and explore more commands and use the power of Codium’s PR agent to enhance your workflow.

Pricing

This is where it gets more interesting. Codium PR agent as I mentioned earlier is open source and can be used in any git-supported platform such as Gitlab, Github, etc. It’s free to use and you only need to configure your openAI API key. It’s way cheaper than GitHub for sure.

Data Privacy

Codium doesn’t store any communication data. Since it’s linked with OpenAI, the communication between your PR instance and openAI is with you and not with Codium.

Conclusion

AI solutions are one of the great productivity contribution elements in software development fields. We want to ensure that the majority of hours of developers are involved in writing good code and the rest is metadata work. Assisting and automating the developer workflow will save so many hours and cost and the Codium PR tool is one of those tools which is a game changer.

Building Email Spam Detection Model in Python

Shahid Shaikh — Thu, 21 Sep 2023 08:21:30 +0000

I am pretty sure everyone who is reading this knows about email spam. This is a very general problem; almost all major email platforms have solved this problem to some extent. But, have you wondered how it actually works? In this article, we are going to explore the same and build and train a Machine Learning model that will detect incoming email and identify it as spam or ham.

Prerequisites

This is going to little a lengthy tutorial so be ready with your coffee and make sure the following software is already installed on your machine:

Python
Jupyter Notebook
Visual Studio Code or Editor of your choice

Here are quick links to the installation page of Python and Jupyter Notebook.

You should also install the Anaconda package that contains the majority of the libraries we need for Machine learning in Python. Click here to go to the downloads page.

I also assume that you know the basics of Machine learning and Python because I won't be digging too deep into basic concepts.

Problem Statement

Build and train a Machine learning model with good accuracy to detect emails as spam or ham (no spam). We should use standard libraries available for Machine learning model building.

Dataset

To train any model, you need a pre-defined dataset that contains good and clean data. By clean I mean it should not have null values, ambiguous data or empty data. For this tutorial, I have identified a clean dataset and you can download it directly by clicking the link below.

Spam Dataset

Code

Let's jump into the code. Open your Jupyter Notebook and import all the libraries we will need.

import numpy as np
import pandas as pd
from sklearn.model_selection  import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn import metrics

Now, import the CSV you just downloaded as a pandas data frame.

emailData = pd.read_csv("spam.csv")
emailData.head(10)

You should have the file data in your notebook similar to this.

Analyze the data and ensure there are no null values using the following code.

emailData.shape
emailData.isnull().sum()
emailData.describe()

Our target variable in the dataset is the Category and we should convert it into a numerical variable for the model. Here we will map it this way

Spam -> 1

Ham -> 0

Here is the code.

emailData['Spam'] = emailData.Category.map({"spam":1, "ham":0})
emailData.drop("Category",axis=1,inplace=True)

Now our data is clean and ready to be used in the model building.

Now our target variable is Spam which will determine whether the e-mail is spam or ham. First, will define X and Y variables.

X = emailData.Message
y = emailData.Spam

X is our input which is email messages in the data file and y is our target variable.

Next, we will split our data in a 70-30 split. 70% of the data will be used for training our model and 30% of it will be used for testing purposes. It's not a set practice and some engineers choose 75-25 split which is also fine.

X_train,X_test,y_train,y_test = train_test_split(X,y,random_state=1,test_size=0.3)

The random state ensures that every time we run the code it will produce the same test and train split.

Now comes the important part, we need to convert the email and the words in those emails as tokens and we also need to avoid stop words to not create confusion. Let's create that using the CountVectorizer package.

vect = CountVectorizer(stop_words='english')
vect.fit(X_train)

Let's transform them into a numerical format for the model to understand better.

X_train_transformed = vect.transform(X_train)
X_test_transformed = vect.transform(X_test)

Let's build a model. We will be building a multinominal Naive Bayes Model which is widely used for text-based prediction.

mnb = MultinomialNB()
mnb.fit(X_train_transformed,y_train)

Now, that our model is instantiated, let's test it with our test data.

y_pred = mnb.predict(X_test_transformed)

Let's check the accuracy score.

metrics.accuracy_score(y_pred, y_test)

For me, it's coming around 0.9856459330143541 i.e 98% accuracy.

Testing the Model with Real Data

Time to test our model with some real data. I copied the following e-mail from my Gmail Inbox to see how it detects spam and ham.

emails=[
    'Sounds great! Are you home now?',
    'Will u meet ur dream partner soon? Is ur career off 2 a flyng start? 2 find out free, txt HORO followed by ur star sign, e. g. HORO ARIES',
    'Hello My Name is Gabriel Robert a top Bank Officer with Standard Bank of South Africa and I am in need of a reliable foreigner to carry out this important deal. An account was opened in my bank by one of my customers a Dutch National from Germany who made a fixed deposit of $11,100,000.00 (Eleven Million, One hundred Thousand United States Dollars) and never show up again and I later discovered that he died with his entire family members on a plane crash that occurred in Libya on the 12th of May 2010 which I can give you a link to that crash if you care. Since nobody is coming for this fund or knows about this fund I want to present a foreigner like you as next of kin to my late client so we can make the claim and you can contact me if you are interested so I can give you more detailed information about this transaction. For the sharing of the money will be shared in the ratio of 50% for me, 40% for you and 10% to cover our expenses after the deal. Now the total amount to be transferred is $12.2 million because of the interest the fund has accumulated for some recent time. Please keep this absolutely confidential and tell me if you are interested but I can assure you 100% risk free as I know how to go about it. Waiting for your urgent reply and call. Thanks.'
]

Here is the code to predict this with our model.

email_1 = vect.transform(emails)
mnb.predict(email_1)

Should give us the following output.

array([0, 1, 1])

Here 0 means ham and 1 means spam. So in my case, it is correct and detected the spam correctly. Do a test with your own e-mail data from your inbox and see how it's reacting to your data.

This article was first published on Shahid's personal blog.

MySQL pagination

Shahid Shaikh — Tue, 20 Jul 2021 15:33:46 +0000

MySQL is one of the most popular database software and most loved as well by software engineers. I have used MySQL in tons of projects and have complete confidence in this software. I am sure if you have a software engineering background, you must have come across MySQL.

MySQL stores data in the database and table format and when the size of data grows it becomes a performance bottleneck to read all of the data. For few thousand records, it’s alright to run a SELECT * query and fetch all the records. It’s not recommended to do it for hundred thousand or million records.

Let’s consider a very simple data record, a user table that contains all the information about users. To fetch the record, you run this query.

SELECT * FROM users;

This returns all the records, you can filter them by providing a WHERE clause but the MySQL engine needs to scan all of the records to find the match.

To avoid this scenario, we use the pagination approach. In the pagination approach, we send back the information in pages or batches instead of the whole at once.

For instance, we send 100 records per page and the application can request more data if requires by asking for the next page data.

To achieve this in MySQL, we use the OFFSET and LIMIT clauses.

The OFFSET clause lets you skip the records and the LIMIT clause lets you limit the record to a certain number.

For example, if we want to read the 50 records per page. We run our query like this.

For page 1

SELECT * FROM users OFFSET 0 LIMIT 50;

For the next page i.e page 2, we tweak the query like this.

SELECT * FROM users OFFSET 50 LIMIT 50;

For the next page i.e page 3, we tweak the query like this.

SELECT * FROM users OFFSET 150 LIMIT 50;

And so on.

This article was first published on Shahid's blog.

Generating PDF Documents on the Fly Using Nodejs and Bull

Shahid Shaikh — Mon, 19 Apr 2021 08:55:30 +0000

PDF documents are commonly used files in the majority of web applications. PDF documents are used in billing invoices, generating transaction reports, drafting the online agreement, etc. If you are building a SaaS then you might end up writing code or using a service to handle generating PDF documents on the fly as required by the system.

In this tutorial, we are going to learn how to generate PDF documents on the fly using Node.js and the Bull queue system.

Generating PDF Documents using Nodejs and Bull

Bull is a Redis-backed queue system built for scalability. To use Bull, you must have Redis key-value database running in your system. You can download and run Redis by following the guide from the official download page of Redis.

Let's begin with the project.

Create new project in Node

To create a new project in Node, create a new folder and name it as you like, open the terminal or command prompt and switch to the folder you created using the cd command.

Run this command to create a new Node project.

npm init --y

This command will generate the boilerplate package.json for your project.

Run this command to install the dependencies required by the project.

npm install --save bull pdfkit

Let's code our project. Our codebase is divided into two important files, first is job.js which is responsible for creating jobs in the queue, and the second file is worker.js responsible for fulfilling the jobs and creating PDF files.

Let's check out each file and codebase first. Here is the job.js code.

const bull = require('bull');
const invoice = require('./invoice');
const queue = new bull('pdf-generation');

function startJob() {
    let invoiceData = invoice.content;
    invoiceData.forEach(async (singleInvoice) => {
        // push data in queue
        let job = await queue.add({
            title: `Generate invoice ${singleInvoice.index}`,
            template: singleInvoice.text,
        }, {'delay': 1000});
    });
}

startJob();

In this file, we are going through all the records present in the invoice.js file. This is for sample record purposes, you can very well use the database records to the same.

Here is the sample content of invoice.js.

module.exports = {
    content: [{
        'index': 1,
        'text': 'Hello Shahid, You are charged $100 this month for services.'
    },{
        'index': 2,
        'text': 'Hello Jack, You are charged $50 this month for services.'
    },{
        'index': 3,
        'text': 'Hello Linda, You are charged $76 this month for services.'
    }]
}

The codebase in job.js retrieve and loop over this records and creates jobs in the queue for each record by having a delay of 1000ms. Let's check out our worker.js code.

const bull = require("bull");
const queue = new bull('pdf-generation');
const pdfKit = require('pdfkit');
const fs = require('fs');

function startProcess() {
    // listen to the queue
    // start processing email
    queue.process((job) => {
        // on each request generate the pdf
        console.log(`Processing Job with id ${job.id}`);
        generatePdfInvoice(job.data);
    });
}

function generatePdfInvoice(data) {
    let doc = new pdfKit;
    doc.pipe(fs.createWriteStream(`${__dirname}/invoice/${data.title}.pdf`));
    doc.fontSize(14).text(data.template, 100, 100);
    doc.end();    
    console.log(`Generated PDF document`);
}

startProcess();
console.log('Worker running');

We are using a node module called pdfkit to generate our PDF documents. The function startProcess() invokes the worker job and it listens to the queue and wait for any messages.

Once it receives a message, we are calling generatePdfInvoice() function with the information received in the message. In the generatePdfInvoice() function, we are creating the PDF record by using the data coming from the queue and writing the PDF file in the invoice folder.

Let's run the code and see it's working.

First, run the job.js code using the following command.

node job.js

Then, run the worker.js file.

node worker.js

Open the invoice folder and check the PDF files.

We have successfully created the PDF file. This is a sample PDF file and you can add as many details as you want including images, tables, etc in your PDF files.

Conclusion

We have studied how to use Node and queue systems such as Bull to design and build systems to generate PDF documents on the fly for business purposes.

This article was first published on codeforgeek.com

What is Bitcoin and How does it work?

Shahid Shaikh — Tue, 16 Mar 2021 12:04:58 +0000

Bitcoin is the world’s first peer-to-peer electronic cash system. Bitcoin offers anonymity, transparency, and no third-party dependent system to exchange value such as Money.

Satoshi Nakamoto invented Bitcoin in 2009, and it’s been hailed as one of the most important inventions of the century.

Bitcoin was invented to solve trust-based dependency in the current commerce system. We as people trust banks, governments, and federal reserves to safeguard money and provide a platform for the common man to trade and exchange using paper currency.

Due to these trust-based models, the control of the economy is in the hands of a few people, who can use it for malicious purposes. I don’t want to go into detail about it because our article is not meant for bashing the banks and governments ;)

However, there was no alternative, and you had to go to the bank to handle your financial needs. Then, Bitcoin arrived as a new and better alternative.

To trade in Bitcoin, you don’t need any bank or government approval. You can send money to your peers using your computer or mobile. It’s as simple as that.

In this article, we will learn in-depth about how Bitcoin works, how each transaction is formed and added to the block, and how it has done more than just solving our money problems.

Note: I highly recommend our readers to print out the Bitcoin
whitepaper written by Satoshi and give it a thorough read. You can download the paper from https://bitcoin.org/bitcoin.pdf

What Is Bitcoin and How Does It Work?

Bitcoin is a peer-to-peer electronic payment system created in 2009 by Satoshi Nakamoto. Bitcoin allows us to send money to anyone in the world, without the need for a central authority such as banks to issue accounts or process payments.

Bitcoin was created as a solution to the current financial system, where a marginal number of large corporations and banks control the financial world and the processing of transactions across the world.

It centralizes the control of finance and commerce and forces people to trust the banks to act responsibly. And like Satoshi said in the P2P foundation.

"The central bank must be trusted not to debase the currency, but the history of fiat currencies is full of breaches of that trust"

Due to this trust issue, a major financial crisis affected the world at the end of 2007. This crisis inspired Satoshi to create Bitcoin.

Bitcoin is a computer program. You can download it precompiled, or you can download the source code from Github and compile it
yourself. Once you run the Bitcoin program on your computer, you become part of the network; you are a peer!

Once your computer becomes a part of the network, the program
will connect to other computers running the same program. Once
connected, computers will start sharing a file and try to sync the files with each other; this file is nothing but a Blockchain. Once your computer is up to date with the Blockchain files, you can start submitting the transaction.

It may raise a question: does everyone need to run the Bitcoin program to send/receive Bitcoin? Well, no. You can download a Bitcoin wallet and start making Bitcoin transactions. Those who wish to become a peer or miners in the network need to run the Bitcoin program

Once your computer is synced with the latest copy of the Blockchain, you are good to go. You can now submit transactions, mine the blocks, read the Blockchain, and do whatever you wish to do.

When a new transaction is submitted in the Blockchain, it is broadcast across the network. Every node then makes a copy of that transaction and adds it to their memory pool.

In the Bitcoin network, at around ~10 minutes of the interval, one of the miners will mine a new block and broadcast it across the network. Every node will then update its Blockchain, and the network will be up to date:

This peer-to-peer nature of the system creates a trust-free, electronic cash system that runs with a solid consensus mechanism.

Before we proceed ahead with the technical aspects of Bitcoin,
let’s understand one of the key issues that Bitcoin solved—double-spending.

What is a Double Spending Issue?

Bitcoin is a peer-to-peer system, so what if two nodes submit the same transaction at the same time? How will the network with no central authority handle such a scenario? It is a serious and groundbreaking issue; let’s learn how Bitcoin solved it.

Bitcoin solved this problem by introducing a memory pool. The
memory pool is a buffer that every node has to maintain. The memory pool contains the transactions that are later written to the Blockchain.

When a node submits a new block, it is broadcast throughout the
network:

When a new block is submitted to the network, every node has to
accept that as a "correct" transaction and remove any conflicting transaction from their memory pool.

As a result, no chance of double spending can occur because it won’t persist in the network if it’s already been added to the block.

This double-spending issue is groundbreaking and opens the way
for many other projects such as Ripple cryptocurrency, which works with the current finance system but offers a solution to the double-spending.

Now, let’s look over the technical aspects of Bitcoin.

Blocks

Block is the data structure that holds transactions in an immutable form. Bitcoin stores users’ transactions in the blocks, and later they form a chain to build a Blockchain network.

As we discussed in the previous sections, the block is generated by the miner by the process of mining. The mining process requires a consensus algorithm to reach an agreement in the network.

Block consists of the block header, which is used by the miner to generate the hash. Block header consists of the following:

Version: The version of the block
Previous block hash: Hash of the last identified block, thus forming a chain
Merkle root: Hash of all of the transactions in the block
Time: Timestamp
Bits: Hexadecimal version of the target value
Nonce: Is a 32-bit random whole number that is adjusted by the miners, so that it becomes a valid number to be used to provide the hash of the block that should be less than the target hash.

The first two fields are self-explanatory, and we have already discussed them in the previous sections. Let’s learn about the Merkle tree.

Merkle Tree

Merkle tree is the data structure in which every leaf node is labeled with the cryptographic hash of the data, and every non-leaf node is labeled with the hash of its child nodes, forming a tree to the root with one hash value.

As you can see in the following image, hash 0-0 and hash 0-1 is the hash value of data block L1 and L2. Later, Hash 0-0 and 0-1 will be hashed together to form Hash 0 and so on until we reach a root hash:

Using a Merkle tree, we can generate a hash of a long series of data in a compact and secure format:

As shown in the preceding screenshot, the Merkle root contains transactions that are hashed together in a way the binary tree is formed.

Merkle tree is formed by hashing the transaction IDs in a pair. Transaction IDs are used if needed, and we can perform a recheck using the transaction ID in a tree.

To generate a Merkle tree, we take a pair of transaction IDs (TXID) and hash them twice using the SHA256 algorithm. The resulting hash is then hashed with another hashed pair twice to generate a single hash. This goes on till we generate a root hash.

Merkle trees are best suited for applications such as Bitcoin, where the size of the data matters. Due to the use of the Merkle tree, the block size is retained to 1Mb max in Bitcoin.

Learn in-depth about the Merkle trees; it’s one of the most frequently asked questions on Facebook, Amazon, Apple, Netflix, Google (FAANG) companies. Start from here.

Let’s learn about the next field in the block header—Bits.

Bits

Bits is the field in the block header that stores the target value in the compact. Target is the value that is required for the miner to generate the block hash that is less than the target. We studied it in the Blockchain core concepts section.

Let’s look at an example.

Here’s what a target value looks like:

0x00000000000000000696f4000000000000000000000000000000000000000000

Storing such long values will require size, and mind you, the target will keep changing in the future.

To solve this, the target is compacted, and it looks like this in the block header field:

0x180696f4

It’s easy to store and takes less size.

Here’s how to decode a bit. It’s divided into two parts. The first two bits after 0x represent the number of 0s in the target value, and the remaining values are appended after 0. The rest of the field is then filled with 0s till it reaches 48 bit.

So, 0x180696f4 value can be written as:

0x----18 times zero---0696f4---rest zeros----

This way, we can decode the bits in the actual target value.

In the next section, we’ll look at the field, the miner used to generate the hash of the block that is lower than the target value.

Nonce

The Nonce is the field in the block header that is used by the miner to generate a hash. Bitcoin uses proof of work consensus algorithms. In proof of work, Miner has to generate a hash that is lower than the target value decided by the network.

Target value starts with several 0s, and it’s really difficult to get such a hash using the SHA256 algorithm on a random basis. So, Nonce comes into the picture, Miner uses the Nonce field to generate a hash, and if it is not lower than the target value, Miner increments the Nonce and tries again.

When the lucky number or Nonce is found, a hash is generated. Miner announces the block in the network, and every other node verifies and makes a copy of it in their Blockchain.

So, Nonce plays a crucial role in Bitcoin mining.

We have studied the block and important block header fields, so let’s move ahead and learn about Bitcoin transactions.

Transactions

When someone tries to transfer Bitcoin from one account to another, a new transaction is formed and added to the block by the miner.

The transaction contains data that is required for proper Bitcoin transfer. Transaction data consists of the amount that the user wants to send, account information such as from and to addresses, transaction information required for the network such as transaction Ids, and so on.

Transaction in Bitcoin contains the following information:

The transaction ID is also referred widely as TXID
Transaction data such as INPUT, OUTPUT, and so on
Fees
Weight
UTXO

Let’s learn about each one of them in brief.

TXID

TXID is a 32 bytes hexadecimal number that is referred to as transaction ID and used to identify the transaction in Bitcoin.

TXID is generated by hashing the transaction data twice using the SHA256 hashing algorithm.

Satoshi Nakamoto sent the first-ever Bitcoin transaction to Hal Finney, and the ID of the transaction is this:

f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16

You can still view this transaction in the Blockchain explorer:

Let’s look over the transaction data required to form the transaction payload.

Transaction Data

Transaction data contains the information required to form INPUT and OUTPUT. Now, what are they? Well, in Bitcoin, when you send someone a few coins, you unlock a few outputs that are required and lock them again with the keys of the receiver so that only they can lock it.

Sounds confusing? Stay with me.

Let’s take an example:

Imagine you have 5 Bitcoins, and you want to send 3 bitcoins to your friend. Here, 5 Bitcoins you hold is your OUTPUT, and it is split into two OUTPUT. One worth 3 Bitcoins and another worth 2 Bitcoins. The first OUTPUT will then be locked with the receiver keys so that they can unlock it. The second OUTPUT is your money, so it is again locked with your credentials for future uses.

In a nutshell, when a new transaction is formed, the whole amount is included in the transaction payload. It’s just whatever is left is being sent back to you as an OUTPUT.

So, this approach of graph-based INPUT and OUTPUT structure provides a history of ownership of Bitcoin.

Fees

Every transaction in Bitcoin pays some amount of money in the form of Bitcoin to the Miner. The remainder of the transaction output is also associated with the fees and paid to the miner for the work they do.

Generally, by adding lucrative fees, there is a high chance that a miner will pick your transaction from the memory pool to add into the next block. However, this is completely optional. If you don’t pay the fee at the moment, it may take some time, but your transaction will be mined by some miners scattered across the world.

Here is the fee associated with the transaction. You can also check the miner award, which is 6.25 Bitcoin at the time of writing this article:

The remainder of the transaction is also paid to the Miner as a fee. Suppose you want to pay someone 5 Bitcoin, and you have around 10 Bitcoin in your wallet, if you don’t claim the remaining 5 Bitcoin in the OUTPUT section of the transaction, it will be paid to the miner. It’s not refundable because, well, Blockchain is immutable! So, double-check before forming a transaction payload programmatically.

Weight

As the name suggests, it refers to the size of the transaction, that is, its weight. After the introduction of weight, the transaction payload is formed by putting the validation part at the end of the payload rather than with each INPUT, as we did earlier.

UTXO

It’s a short version of Unspent Transaction Output. When a new transaction is formed and new OUTPUT is generated, the one which is spent or sent to other users cannot be spent again. However, the OUTPUTs left or not spent can be used in future transactions.

These unspent outputs are called UTXOs.

UTXO is used to calculate the balance of a particular user and verify whether they have the amount to spend. UTXOs are a key part of forming new transactions and preventing double-spending.

These are the key factors that are involved in the Bitcoin transaction.

I hope you understood the concepts and the reasoning behind the technical changes done to make Bitcoin and Blockchain what they are now. Let's learn and understand another important and key technical aspect of Bitcoin - Cryptography.

Addresses and Keys

In order to send and receive Bitcoins, you need an address and to be specific a secure address. Bitcoin uses pair of public and private keys. The public and private keys play a crucial role in making a Bitcoin transaction secure and anonymous. The private key is generated by using a long random number and then converted to hexadecimal format. The public key is derived by using the private key.

The compressed version of a public key is also known as an address and you can share this with anyone publicly. Anyone around the globe can send you Bitcoin to your address. In order to send Bitcoin to other people, you need their public key and your private key. Your private key is a secret key and you must never share it with anyone. Each Bitcoin transaction is signed with the sender's private key along with the receiver's public key.

For example, take a look at the private and public key pair shown below:

Private key: ef235aacf90d9f4aadd8c92e4b2562e1d9eb97f0df9ba3b508258739cb013db2
Public key: 02b4632d08485ff1df2db55b9dafd23347d1c47a457072a1e87be26896549a8737
Address: 1EUXSxuUVy2PC5enGXR1a3yxbEjNWMHuem

There is no way to generate the same private key again and there is no way to re-generate the private key from the public key. This makes the private key a crucial part of the system. If you lose your private key, you lose your Bitcoins.

This article is referred from the Book called Building decentralized blockchain applications.Consider a purchase if you like our content. (PS: I am author of this book)

Awesome Node.js Tools, Libraries and Resources

Shahid Shaikh — Thu, 31 Dec 2020 13:18:03 +0000

Node.js is equipped with a rich ecosystem of tools, libraries, and frameworks to help developers build their application faster, safer, and follow the standard industry guidelines on software development.

While there are hundreds of thousands of packages registered in the official node package manager repository, we have shortlisted a few based on the reputation and usability for a day to day software development.

Let's jump in directly and check out some of the awesome Node.js tools, libraries, and frameworks. If you are new to Node, you can read our entry-level complete Node.js tutorial for beginners to get started.

Web frameworks

Here are some of the popular and widely used web frameworks available in the Node.js ecosystem.

Express - A complete web application framework to develop various types of web applications such as Web server, API server,etc. We built this site on top of Express framework and this article is served by an Express router :)
Koa - A expressive framework designed by the team behind Express. Koa aims to provide enjoyable experience to developers while building a web application.
Hapi - a simple and secure framework to build web applications.
Sails.js - MVC (Model - view - controller) framework for Node.js.
Fastify - Fastify is a web framework entirely focused on speed. It is one of the fastest framework in the Node.js ecosystem.
Seneca - It's a tool to write microservices based application in Node.
Molecular - A mature, fast and powerful microservices framework for Node.js.
AdoniJS - is a framework to write micro-services easily.

HTTP and Networking Stuff

Node.js is widely used in building networking applications. There will come a scenario where you as a developer asked to integrate third-party APIs such as payment API, invoice API, etc. To handle these tasks, you can use the libraries mention below.

Axios - A widely used HTTP client for Node as well as the browser.
got - A simple and lightweight interface to make HTTP calls.
HTTP proxy - a very useful tool to build a proxy server effortlessly especially in a microservice architecture.
HTTP fake backend - a huge time saver tool. Allows you to create a fake backend server for the test and demo apps.
download - utility to download files from a URL and save it in file system.
getmac - get the MAC address of your computer.

Database drivers and ODM/ORM

There is really no need to mention how important the database is for your application. Node.js can be integrated with almost every popular database.

MySQL - A pure JavaScript based driver to integrate with MySQL database. We have written a detailed tutorial about Node.js and MySQL.
MongoDB - A MongoDB database driver. Refer this tutorial to learn how to use this driver to connect to MongoDB.
Mongoose - A MongoDB object data model i.e ODM.
PostgreSQL - A database client for PostgreSQL database.
Redis - A client for redis key value database.
LevelUP - A levelDB database.
Couchbase - A client for a couchbase database.
Waterline - A database agnostic that can interact with one or more database.

Logging

Logging is an essential part of any web application. Node.js has some of the best logging library available. I personally uses winston a lot in my projects. It has rich set of plugins to extend the logging feature as per the requirenments.

winston - Asynchronous multi-transport logging library.
pino - Fast logging library.
storyboard - a real time and colorful log.

Documentation

The only thing which almost every developer hate is creating a documentation. I know, I do too. But, it's an important thing to create a well written documentation as you write your beautiful code.

JSDoc - API documentation generator similar to JavaDoc.
documentation - library with modern JavaScript support.
ESDoc - documentation generator with ES2015 support.

Control flow

Node.js is asynchrnous in nature and it's little different than what we used to in high level languages such as Java, C++,etc. We can leverage libraries mentioned below to make our life little bit easier while writing Node.js application.

async - a popular library to write asynchronous code in JavaScript using callbacks.
Bluebird - a library to handle and write asynchronous code using promises.

Streams

get-stream - library to get a stream as a buffer or string.
multistream - combine multiple steams into a single stream.
into-stream - convert existing buffer, array, string into a stream.

Real time

Node.js can be used to build a real time systems. I have used these library in some of the projects and they are pretty good.

Socket.io - A really useful library to develop real time web application.
uWebSockets - A library that support event based real time features.
deepstream.io - A scalable real time framework to develop advanced application like games,etc.
MQTT - A pubsub library to develop applications with lightweight communication.

Authentication and Authorization

When you are building any web application, the most common part is going to be user authentication. You don't need to develop the modules again, Node.js libraries will cover it for you. The libraries listed below are tried and test library in thousands of projects.

Passport - A widely used authentication library that support wide range of SSO.
Grant - A middleware for web frameworks such as Express, koa, and Hapi.
node-casbin - Authorization library with ACL, RBAC and ABAC access control support.

Email

You can use the libraries mentioned below to send an email from your application.

Nodemailer - A popular and widely used library to send emails.
emailjs - A simple and useful library to send text/html emails with attachment.
MJML - Markup based to create responsive emails.

Job Queues and Message queues

Job quues and message queues allows you to develop complex applications and schedule your job based on the requirenment.

bull - extensible job and message queue. We use bull in Codeforgeek.
agenda - Job scheduler using mongodb as a backend.
node-resque - Job queue based on Redis.
rsmq - Message queue based on Redis.
sqs-consumer - library to handle amazon simple queue service in your application.

Process Management

Node.js as a process is tricky to manage. Thanks to the libraries mentioned below especially PM2, managing a Node process is piece of cake.

PM2 - a popular and widely used process manager for Node. We use PM2 in codeforgeek.
nodemon - monitor changes in your file and restart the server on change detection.
supervisor - process manager which restarts the server upon crash.

Static Site Generator

Static sites are pretty popular now a days. With the adoption of gatsby and Netlify, things are moving towards the right direction. You can use the libraries mentioned below to create the static site using Node.

Wintersmith - Multi-platform static site generator.
Assemble - Static site generator with a grunt extension.
DocPad - static site generator with wide range of plugin ecosystem.

Content Management System

Node.js has been used to develop some of the best CMS such as Ghost. You can any libraries and platforms mentioned below to create your CMS and manage your content.

Ghost - widely popular blogging and content management framework and platform.
Hexo - Simple yet powerful blogging framework.
nodeBB - forum platform in Node.js.
KeystoneJS - content management system built on Express and MongoDB.
Strapi - Headless content management system to build API's.
ButterCMS - Headless content management system.

Hardware

You can use Node in hardware devices as well. I have personally used Node in raspberryPI and it works great. You can use libraries mentioned below to build a DIY hardware projects.

Johnny-Five - The JavaScript Robotics Programming Framework.
Node Serialport - Access serial port with Node. Works on cross-platform.
USB - Node library for communicating with USB devices.
onoff - Supports RaspberryPI communication.
GPS - Library to handle GPS receiver communication.

Miscellaneous

Electron - Build a cross-platform desktop application using Node and JavaScript.
OpenCV - Binding for OpenCV computer vision library.
nconf - Nodejs configuration file management.
Cheerio - HTML DOM parser for Node and JavaScript.
Faker.js - Generate huge amounts of fake data.

Conclusion

I would highly recommend you to bookmark this article. We will upgrade this article based on the suggestion and feedback from all of you. If you like to add/remove something from the list, please let me know in the comments or you can make a pull request here.

This article was first published at https://codeforgeek.com/awesome-node-js-tools-libraries-and-resources/

A Guide to Securing Node.js Applications

Shahid Shaikh — Wed, 30 Dec 2020 07:59:31 +0000

The one thing that developers tend to considers at the end of the development cycle is the “security” of the application. A secure application is not a luxury, it’s a necessity. You should consider the security of your application at every phase of the development such as architecture, design, code, and finally the deployment.

In this tutorial, we are going to learn ways to secure our Node.js application. Let’s dive in.

Data Validation – Never Trust Your Users

You must always validate or sanitize the data coming from the user or other entity of the system. The bad validation or no validation at all is a threat to the working system and can lead to a security exploit. You should also escape the output. Let's learn how to validate the incoming data in Node.js. You can use a node module called validator to perform the data validation. For example.

const validator = require('validator');
validator.isEmail('foo@bar.com'); //=> true
validator.isEmail('bar.com'); //=> false

You can also use a module called joi (recommended by Codeforgeek) to perform the data/schema validation. For example.

  const joi = require('joi');
  try {
    const schema = joi.object().keys({
      name: joi.string().min(3).max(45).required(),
      email: joi.string().email().required(),
      password: joi.string().min(6).max(20).required()
    });

    const dataToValidate = {
        name: "Shahid",
        email: "abc.com",
        password: "123456",
    }
    const result = schema.validate(dataToValidate);
    if (result.error) {
      throw result.error.details[0].message;
    }    
  } catch (e) {
      console.log(e);
  }

SQL Injection Attack

SQL injection is an exploit where malicious users can pass unexpected data and change the SQL queries. Let's understand with the example. Assume your SQL query looks like this:

UPDATE users
    SET first_name="' + req.body.first_name +  '" WHERE id=1332;

In a normal scenario, you would expect that this query will look like this:

UPDATE users
    SET first_name = "John" WHERE id = 1332;

Now, if someone passes the first_name as the value shown below:

John", last_name="Wick"; --

Then, your SQL query will look like this:

UPDATE users
    SET first_name="John", last_name="Wick"; --" WHERE id=1001;

If you observe, the WHERE condition is commented out and now the query will update the users table and sets every user’s first name as “John” and last name as “Wick”. This will eventually lead to system failure and if your database has no backup, then you’re doomed.

How to prevent SQL Injection attack

The most useful way to prevent SQL injection attacks is to sanitize input data. You can either validate every single input or validate using parameter binding. Parameter binding is mostly used by the developers as it offers efficiency and security. If you are using a popular ORM such as sequelize, hibernate, etc then they already provide the functions to validate and sanitize your data. If you are using database modules other than ORM such as mysql for Node, you can use the escaping methods provided by the module. Let's learn by example. The codebase shown below is using mysql module for Node.

var mysql = require('mysql');
var connection = mysql.createConnection({
  host     : 'localhost',
  user     : 'me',
  password : 'secret',
  database : 'my_db'
});

connection.connect();

connection.query(
    'UPDATE users SET ?? = ? WHERE ?? = ?',
    ['first_name',req.body.first_name, ,'id',1001],
    function(err, result) {
    //...
});

The double question mark is replaced with the field name and the single question mark is replaced with the value. This will make sure that input is safe. You can also use a stored procedure to increase the level of security but due to lack of maintainability developers tend to avoid using stored procedures. You should also perform the server-side data validation. I do not recommend you to validate each field manually, you can use modules like joi.

Typecasting

JavaScript is a dynamic typed language i.e a value can be of any type. You can use the typecasting method to verify the type of data so that only the intended type of value should go into the database. For example, a user ID can only accept the number, there should be typecasting to ensure that the user ID should only be a number. For example, let's refer to the code we shown above.

var mysql = require('mysql');
var connection = mysql.createConnection({
  host     : 'localhost',
  user     : 'me',
  password : 'secret',
  database : 'my_db'
});

connection.connect();

connection.query(
    'UPDATE users SET ?? = ? WHERE ?? = ?',
    ['first_name',req.body.first_name, ,'id',Number(req.body.ID)],
    function(err, result) {
    //...
});

Did you notice the change? We used Number(req.body.ID) to ensure that ID is always the number. You can refer to this beautiful article by a fellow blogger to understand typecasting in depth.

Application Authentication and Authorization

Sensitive data such as passwords should be stored in the system in a secure way that malicious users don't misuse sensitive information. In this section, we will learn how to store and manage passwords which are quite generic, and pretty much every application has passwords in some way in their system.

Password Hashing

Hashing is a function that generates a fixed-size string from input. The output from the hashing function cannot be decrypted hence it's "one-way" in nature. For data such as passwords, you must always use hashing algorithms to generate a hash version of the input password string which is a plaintext string.

You might be wondering that if the hash is a one-way string then how come attackers gain access to passwords?

Well, as I mentioned above, hashing takes an input string and generates a fixed-length output. So attackers take a reverse approach and they generate the hashes from the general password list, then they compare the hash with the hashes in your system to find the password. This attack is called lookup tables attack.

This is the reason why you as an architect of the system must not allow generic used passwords in your system. To overcome this attack, you can something called "salt". Salt is attached to the password hash to make it unique irrespective of the input. Salt has to be generated securely and randomly so that it is not predictable. The Hashing algorithm we suggest you is BCrypt. At the time of writing this article, Bcrypt has not been exploited and considered cryptographically secure. In Node.js, you can use bcyrpt node module to perform the hashing.

Please refer to the example code below.

const bcrypt = require('bcrypt');

const saltRounds = 10;
const password = "Some-Password@2020";

bcrypt.hash(
    password,
    saltRounds,
    (err, passwordHash) => {

    //we will just print it to the console for now
    //you should store it somewhere and never logs or print it

    console.log("Hashed Password:", passwordHash);
});

The SaltRounds function is the cost of the hash function. The higher the cost, the more secure hash would be generated. You should decide the salt based on the computing power of your server. Once the hash is generated for a password, the password entered by the user will be compared to the hash stored in the database. Refer to the code below for reference.

const bcrypt = require('bcrypt');

const incomingPassword = "Some-Password@2020";
const existingHash = "some-hash-previously-generated"

bcrypt.compare(
    incomingPassword,
    existingHash,
    (err, res) => {
        if(res && res === true) {
            return console.log("Valid Password");
        }
        //invalid password handling here
        else {
            console.log("Invalid Password");
        }
});

Password Storage

Whether you use the database, files to store passwords, you must not store a plain text version. As we studied above, you should generate the hash and store the hash in the system. I generally recommend using varchar(255) data type in case of a password. You can opt for an unlimited length field as well. If you are using bcrypt then you can use varchar(60) field because bcrypt will generate fixed size 60 character hashes.

Authorization

A system with proper user roles and permission prevents malicious users to act outside of their permission. To achieve a proper authorization process, proper roles and permissions are assigned to each user so that they can do certain tasks and nothing more. In Node.js, you can use a famous module called ACL to develop access control lists based on authorization in your system.

const ACL = require('acl2');
const acl = new ACL(new ACL.memoryBackend());
// guest is allowed to view blogs
acl.allow('guest', 'blogs', 'view')
// check if the permission is granted
acl.isAllowed('joed', 'blogs', 'view', (err, res) => {
    if(res){
        console.log("User joed is allowed to view blogs");
    }
});

Checkout the acl2 documentation for more information and example code.

Bruteforce Attack Prevention

Bruteforce is an attack where a hacker uses software to try different passwords repetitively until access is granted i.e valid password is found. To prevent a Bruteforce attack, one of the simplest ways is to wait it out approach. When someone is trying to login into your system and tried an invalid password more than 3 times, make them wait for 60 seconds or so before trying again. This way the attacker is going to be slow and it's gonna take them forever to crack a password.

Another approach to preventing it is to ban the IP that is generating invalid login requests. Your system allows 3 wrong attempts per IP in 24 hours. If someone tries to do brute-forcing then block the IP for 24 hours. This rate-limiting approach is been used by lots of companies to prevent brute-force attacks. If you are using the Express framework, there is a middleware module to enable rate-limiting in incoming requests. Its called express=brute.

You can check the example code below.

Install the dependency.

npm install express-brute --save

Enable it in your route.

const ExpressBrute = require('express-brute');
const store = new ExpressBrute.MemoryStore(); // stores state locally, don't use this in production
const bruteforce = new ExpressBrute(store);

app.post('/auth',
    bruteforce.prevent, // error 429 if we hit this route too often
    function (req, res, next) {
        res.send('Success!');
    }
);
//...

The example code is taken from express-brute module documentation.

Secure Transmission using HTTPS

It is 2021 and you must use HTTPS to send your data and traffic over the internet securely. HTTPS is an extension of the HTTP protocol with secure communication support. By using HTTPS, you can make sure that the traffic and your user's data over the internet is encrypted and safe.

I am not going to explain how HTTPS works in the detail here. We are going to focus on the implementation part of it. I highly recommend you to use LetsEncrypt to generate the SSL certificates for all of your domain/subdomain.

It's free and runs a daemon to update SSL certificates every 90 days. You can learn more about LetsEncrypt here. You can opt for a domain-specific certificate or a wildcard certificate if you have multiple subdomains. LetsEncrypt supports both.

You can use LetsEncrypt for both Apache and Nginx based web servers. I highly recommend performing SSL negotiations in the reverse proxy or at the gateway layer because it is a heavy computing operation.

Session Hijacking Prevention

The session is an important part of any dynamic web application. Having a secure session in the application is a must for the users and systems safety. A session is implemented using cookies and it must be kept secure to prevent session hijacking. The following is a list of the attributes that can be set for each cookie and what they mean:

secure - this attribute tells the browser to only send the cookie if the request is being sent over HTTPS.
HttpOnly - this attribute is used to help prevent attacks such as cross-site scripting since it does not allow the cookie to be accessed via JavaScript.
domain - this attribute is used to compare against the domain of the server in which the URL is being requested. If the domain matches or if it is a sub-domain, then the path attribute will be checked next.
path - in addition to the domain, the URL path that the cookie is valid for can be specified. If the domain and path match, then the cookie will be sent in the request.
expires - this attribute is used to set persistent cookies since the cookie does not expire until the set date is exceeded

You can use express-session npm module to perform session management in the Express framework.

const express = require('express');
const session = require('express-session');
const app = express();

app.use(session({
  secret: 'keyboard cat',
  resave: false,
  saveUninitialized: true,
  cookie: { secure: true, path: '/'}
}));

You can learn more about Express session handling here.

Cross Site Request Forgery (CSRF) Attack Prevention

CSRF is an attack where that manipulates a trusted user of a system to execute unwanted malicious actions on a web application. In Node.js, we can use csurf module to mitigate CSRF attack. This module requires either express-session or cookie-parser to be initialized first. You can check out the example code below.

const express = require('express');
const cookieParser = require('cookie-parser');
const csrf = require('csurf');
const bodyParser = require('body-parser');

// setup route middlewares
const csrfProtection = csrf({ cookie: true });
const parseForm = bodyParser.urlencoded({ extended: false });

// create express app
const app = express();

// we need this because "cookie" is true in csrfProtection
app.use(cookieParser());

app.get('/form', csrfProtection, function(req, res) {
  // pass the csrfToken to the view
  res.render('send', { csrfToken: req.csrfToken() });
});

app.post('/process', parseForm, csrfProtection, function(req, res) {
  res.send('data is being processed');
});

app.listen(3000);

On the web page, you need to create a hidden input type with the value of the CSRF token. For example.

<form action="/process" method="POST">
  <input type="hidden" name="_csrf" value="{{csrfToken}}">

  Favorite color: <input type="text" name="favoriteColor">
  <button type="submit">Submit</button>
</form>

In the case of AJAX requests, you can pass the CSRF token in the header.

var token = document.querySelector('meta[name="csrf-token"]').getAttribute('content');
  headers: {
    'CSRF-Token': token
  }

Denial of Service

Denial of service or DOS is a type of attack where attackers tried to bring down the service or make it inaccessible to users by disrupting the system. The attacker generally flooded the systems with lots of traffic or requests which in turn increases the CPU and memory load leading to a system crash. To mitigate DOS attacks in your Node.js application, the first step would be the identification of such an event. I highly recommend these two modules to be integrated into the system.

Account lockout - After n number of failed attempts, lock the account or IP address for a period of time (say 24h?)
Rate limiting - Limit the users to request the system n number of times within a specific period, for example, 3 requests per minute from an individual user

The Regular expression Denial of service attack (ReDOS)is a type of DOS attack where the attacker exploits the regular expression implementation in the system. Some regular expression takes heavy computing power to execute and the attacker can exploit it by submitting requests that involve regular expression in the system which in turns increases the load on the system leading to system failure. You can use software like this to detect the dangerous regular expressions and avoid using them in your system.

Dependencies Validation

We all use tons of dependencies in our projects. We need to check and validate these dependencies as well to ensure the security of the overall project. NPM already has an audit feature to find the vulnerability of the project. Just run the command shown below in your source code directory.

npm audit

To fix the vulnerability, you can run this command.

npm audit fix

You can also run the dry run to check the fix before applying it to your project.

npm audit fix --dry-run --json

HTTP Security Headers

HTTP provides several security headers that can prevent commonly known attacks. If you are using the Express framework then you can use a module called helmet to enable all security headers with a single line of code.

npm install helmet --save

Here is how to use it.

const express = require("express"); 
const helmet = require("helmet");  
const app = express(); 
app.use(helmet());  
//...

This enables the following HTTP headers.

Strict-Transport-Security
X-frame-Options
X-XSS-Protection
X-Content-Type-Protection
Content-Security-Policy
Cache-Control
Expect-CT
Disable X-Powered-By

These headers prevent malicious users from various types of attacks such as clickjacking, cross-site scripting, etc.

Author's blog: https://shaikhshahid.com

Redis Streams for Beginners

Shahid Shaikh — Tue, 22 Sep 2020 18:41:27 +0000

Redis is the most popular key-value database store. Redis is used in the majority of large scale applications.

While building large scale distributed systems, we often end up in a scenario where we have to perform the huge amount of data ingestion or data transfer from one system to another. The only feasible way to perform this is through streaming. Distributed engineers generally use streaming software such as Apache Kafka to perform data streaming and ingestion.

Redis with the support of version 5.0 offers an innovative way to manage streams. Redis Stream is a built-in feature and it offers a data structure that developers can use to perform data ingestion, data consumption, creates a data channel between producer and consumers, and so on.

In this tutorial, we are going to explore how to use Redis streams and how you as a distributed systems engineer can incorporate Redis streams in your architecture.

Redis streams capabilities

Redis is used in various use cases across the system. This includes content caching, real-time analytics, message brokers, etc.

While Redis already provides a publish/subscribe feature, with the release of Redis streams, Redis offers the following capabilities:

Collect large amounts of data.
Create a data channel between many consumers and producers.
Data ordering is effectively managed even though the rate of data ingestion and data consumption is different.
Offline data persistence support.
Asynchronous communication between producer and consumer.
Easily scalable to large number producers and consumers.

Like I mentioned earlier, Redis streams are built-in into Redis. You don’t need to perform extra installation or management of heavy software. It’s right into Redis.

Let’s jump right into Redis streams and learn how to use it.

Using Redis Streams

Redis Streams is an append-only log-based data structure. Redis streams offer commands to add data in streams, consume streams, and manage how data is consumed.

Redis streams can have one to one communication or one to many or many to many communication streams between producers and consumers. Redis streams also offer consumer groups so that they can consume only the information that they are supposed to receive.

Let’s use Redis streams commands. I am assuming that you have Redis version 5.0 or above already installed in your system. If not, please navigate to the Redis download page to download the latest version of Redis.

Once you have Redis installation sorted out, start the Redis server if you are using it in your system using the following command.

redis-server

Some installers already run the Redis server in the background, so if you get some error, don’t worry about it.

Now, let’s start the Redis command line shell using the following command.

redis-cli

Let’s add some data in Redis streams. The default and easy way are to run this command.

XADD mystream * name Shahid

Here, mystream is the name of the stream. Name is the key and Shahid (My name) is the value.

When you run this command, Redis will generate the ID for your stream data.

We can also add our own ID while adding the records. For example,

XADD mystream 10000000 name Shahid

However, you should rely on a system-generated unique ID.

Let’s consume data from the Redis streams.

To read everything from the start of the stream to the end, we can use the following command.

XREAD STREAMS mystream 0

Here we are giving the name of the stream and start index i.e 0. This command will return all the records present in the stream.

We can also limit the records by providing the record count.

XREAD COUNT 100 STREAMS mystream 0

We can also iterate through the stream using the XRANGE command.

For example,

XRANGE mystream 1518951123450-0 1518951123460-0

The command shown above will read the record from the stream between the ID mentioned in the command.

If we don’t know the ID of the records, we can pick the number of records using the COUNT operator.

XRANGE mystream - + COUNT 10

The command shown above will return the record from 10 records iterating from the beginning of the stream.

Redis streams offer more commands and ways to manage streams. You can learn more about it on the documentation page here.