DEV Community: Saurabh Dashora

Why the Cloud Revolution is just getting started?

Saurabh Dashora — Mon, 20 Feb 2023 10:59:58 +0000

When cloud computing burst onto the scene in 2006 with the launch of AWS, it would have been hard to imagine how big of a thing it would eventually become.

But more than 15 years later, cloud computing has come a long way. And yet, in my view, it is only just getting started towards realizing its true potential.

Why do I think this way?

Recently, I came across this Gartner study that contained a couple of mind-boggling facts:

More than 85% of organizations will embrace a cloud-first principle by 2025
Over 95% of new digital workloads in 2025 will be deployed on cloud-native platforms, up from 30% in 2021

Of course, numbers can be spurious. But when you talk about 85% of organizations and 95% of all new digital workloads, it is definitely a lot. Even if the figures are off the mark by some points, they are still huge.

My curiosity about the matter was piqued and I started digging more into the potential trends that might fuel this anticipated growth. Naturally, as a software developer, I’m always interested in knowing where the industry is headed because that’s how we can prepare and hope to keep ourselves relevant.

After doing some reading up, I have formed an initial idea of the broad trends that are driving cloud computing and will continue to do so in the coming years.

However, before I share them with you, I wanted to make some points about cloud computing that can help us understand the trends in a much better way.

You may or may not agree with the points I make. Either way, I’d love it if you share your views in the comments section below.

What made the Cloud so popular?

I believe that the seeds of the future are laid in the past. This must hold true for cloud computing as well.

But what made the cloud so popular?

In my view, cloud computing democratized the ability to build applications with world-class infrastructure.

No longer do you need to be a multi-billion dollar organization with an army of engineers to create applications used by millions of people. Even a startup working out of a garage can do it.

So, what was stopping the same thing from happening in the pre-cloud era?

For starters, the pre-cloud era could also be labelled as the on-premise era. This meant that organizations typically managed their own IT infrastructure and resources.

For example, if you wanted to create an application and make it available to the world in the pre-cloud era, you had to purchase, install and maintain hardware and software in-house.

This arrangement had a couple of big technical implications:

Management of IT infrastructure such as servers, storage and networking lay solely on the shoulders of the organization’s workforce. I still vividly remember the anecdote of one of my seniors about the earlier days when he had to even fix network cables when a broken connection brought their application down in the middle of the night and the network vendor was not available.
The IT systems were not scalable based on demand since organizations were limited by physical resources. If there was an expectation of higher demand, you had to go out and buy more resources. For that to happen, you had to be really good at predicting demand else you would be incurring extra costs for no reason.

Of course, when are higher-ups in organizations worried about tech issues unless there was a threat to the company’s bottom line?

However, in this case, there were threats.

For starters, on-premise computing is a costly business. You need a significant up-front investment in hardware and software to build a data centre.

Initially, big companies loved this situation. It was like a huge barrier to entry for smaller players. However, once the genie was out-of-the-bottle and some cloud offerings came on the scene, the huge cost associated with on-premise computing became a liability.

Suddenly, the army of engineers hired to just keep the infrastructure running started to look like money-guzzling machines.

In more disruptive industries, startups with a skeleton crew of software engineers were leap-frogging established players by using the initial cloud tools to drive faster innovation and reducing time-to-market. This meant a loss of market share and growth opportunities for the big companies.

Of course, all of this didn’t happen in a single day, a month or even a year. But slowly and steadily, large organizations also started to steer their ships in the direction of cloud computing.

Once that happened, there was no looking back.

With more investment coming into the business, cloud computing took off with multiple vendors entering the field. Companies like Google and Microsoft started to compete with Amazon for a pie of this incredibly lucrative business model.

It was like discovering an untapped oil field right next to your front door.

So, where are things headed?

Predicting the direction of a particular technology can be a fool’s errand. Even in 2006, not even the creators of AWS would have predicted the kind of growth they have seen.

But, of course, it’s important to make an educated guess so that we are better prepared for what’s coming in the next few years.

Here are a few broad trends I’m tracking:

Hybrid Cloud Adoption

This one’s a biggie as it is driven mostly by the large organizations that run the world. Think of big banks, government organizations and mega-corporations.

The trend is largely driven by the growing regulatory and legal requirements about data and the increase in privacy concerns all across the world.

In a hybrid cloud setup, companies want to keep a mix of capabilities across external cloud platforms as well as in-house setups. The idea is to use the public platforms for new innovative products but keep the core-business capabilities or data on in-house data centres so that they don’t run foul of government regulations.

Since it involves big money, I feel hybrid cloud adoption is only going to grow. Already, big cloud providers are rolling out products to support this vision.

Red Hat has been offering its flagship OpenShift platform as an on-premise solution for many years now.
Microsoft has launched Azure Arc to cater to hybrid and multi-cloud requirements.
Google has launched Anthos - a platform that promises a single and consistent way of managing Kubernetes workloads across the public and on-premises cloud.

Multi-Cloud Adoption

“Don’t keep all your eggs in one basket”

This adage is so wise that organizations are increasingly exploring the use of a multi-cloud approach.

A large part of the multi-cloud adoption is driven by risk mitigation. For example, fin-tech organization Form3 was compelled to go for a multi-cloud setup when the regulators questioned them about the portability of their platform in case AWS went down.

However, some of the shift to multi-cloud is also a result of increased competition and service offerings by different cloud vendors. Even beyond the Big 3, there are dozens of other cloud providers providing all manner of cloud services to lure customers with cost or features.

Organizations have been spoilt for choice and are trying to get the best ROI for every piece of their infrastructure.

I feel this trend is going to accelerate in the future.

The difficulties of managing a multi-cloud setup could have curtailed this movement. However, instead of getting bogged down, the demand for multi-cloud and hybrid-cloud setups has spurred a number of new trends such as the rise of infrastructure-as-code tools and the concept of platform engineering. I will discuss more of them in upcoming posts.

Serverless Computing

One of the main factors that worked in favour of the cloud in the initial days was cost savings. The idea that you could even launch a product with close-to-zero costs was hard to beat and created a tremendous rush for cloud adoption.

In my view, serverless has the potential to make even traditional cloud offerings appear costly. Though a few years have passed since serverless options were launched by most major cloud providers, I feel that we are only at the beginning of the serverless revolution.

Since serverless computing allows companies to run code without even provisioning or managing cloud servers, it is extremely lucrative for organizations that want to save on costs and move faster.

With the tremendous rise in the number of SaaS startups and the existence of an inflationary environment with rising interest rates, the cost of running your system is a big issue.

Organizations are looking to achieve product-market fit without burning through too much cash and serverless computing seems like a good deal with its pay-per-use model and little to no maintenance expenditure.

AI-Driven Cloud

Apart from cloud computing, the last decade or so has also seen another major trend spread like wildfire - the rise of machine learning and artificial intelligence.

As AI seeps into more and more areas and supports real requirements, it is already promising to augment cloud services in really interesting ways.

For example, AI-driven cloud services can potentially make autonomous decisions on when to scale up or down based on an intuitive understanding of demand rather than fixed rules.

Again, this boils down to monetary benefits with the promise of better cost utilization.

Of course, we can only hope that one of these services doesn’t turn into Skynet any time soon!

Either way, I’ll be extremely interested in keeping an eye out for this developing trend.

Containers as a Service

Containers on the cloud started off quite early with Amazon launching ECS. Of course, managing a bunch of containers isn’t the easiest thing out there.

However, the surging popularity of Kubernetes has changed the landscape of container orchestration. And within no time, all major and minor cloud providers are offering managed Kubernetes services.

This is one area where big and small organizations are lapping up the opportunity. After all, everyone wants to reap the benefits of containerization without acquiring the headache of managing them.

As developers, it is definitely important to keep abreast of this trend.

That’s it for now!

In the end, I feel that we are living in interesting times when it comes to cloud computing.

The technology is at the right level of maturity where it has become mainstream enough to have a large base of innovation. However, it is also not so dormant that things become boring and static.

To top it off, cloud computing is also producing other trends in areas such as microservices architecture, DevOps, infrastructure-as-code and platform engineering.

As we learn more about the cloud, we also will be learning a lot about these other secondary trends that have the potential to become even more important. I will definitely be talking about them a lot more in the upcoming days and weeks.

If you liked today’s post, consider sharing it with friends and colleagues.

Do you want to stay updated and relevant to the latest trends in technology?

Do you also want to get actionable advice about cloud and infrastructure topics?

If yes, subscribe to the ProgressiveCoder newsletter for FREE.

5 Must-Ask questions before you implement Caching

Saurabh Dashora — Wed, 08 Feb 2023 08:03:24 +0000

There are only two hard things in Computer Science: cache invalidation and naming things.

The above statement by Phil Karlton has acquired some sort of legendary status in software development circles. And not without reason.

Caching has the potential to be the most important piece in your app development puzzle. It solves a bunch of important problems in modern app development such as :

performance
scalability
cost
availability

But despite all the benefits, you must not decide to use it blindly.

Any approach is beneficial only when it provides some value over the current situation.

Before deciding to implement caching, ask yourself the below questions.

1 - Is the operation I’m trying to cache slow?

Caching the result of an operation is beneficial only if the operation is really slow or resource-intensive.

For example, let’s say you are trying to use an external Weather API to retrieve some information. The Weather API may be slow or it may be expensive to use due to usage limits (in other words, resource-intensive).

In this case, you would do well to implement a cache to store the results of the Weather API query. When a user makes a query, you can first check if the data is available in the cache and call the slow and costly Weather API only when needed.

As a rule of thumb, always check whether you are trying to access a slow external API or a database. If yes, only then consider the use of caching. Otherwise, you would end up caching for no benefit and additional complexity.

2 - Is the cache actually faster?

Don’t cache just for the sake of caching!

The cache must be able to store and retrieve faster than the original source. Alternatively, it should consume fewer resources.

However, sometimes it is not immediately clear if caching will be advantageous. To make a decision, try to set up a test environment where you can simulate a high volume of traffic. Run tests with and without the cache and compare the results. If the performance improves due to caching, then and only then go for a caching-based solution.

Ultimately, there should be some quantitative advantage of using cache.

3 - Is the data I’m trying to cache dynamic?

Suppose your cache stores the results of a search query. When a user makes a new search query, you retrieve the results and also store them in the cache. On subsequent requests for the same query by other users, you return the cached results.

While caching such data, ask yourself how long until the cached result becomes stale.

If the cached data becomes stale very frequently and you have to invalidate it, you might not get sufficient advantage from caching. Items that don’t change from request to request are better candidates for caching.

4 - Is the data frequently accessed?

Let’s consider our earlier example of an e-commerce platform.

If you think there is a popular product in your catalog, its product page may receive a ton of requests. If the details are fetched from the database every single time, your application may perform poorly due to overloading. This is an ideal scenario to explore the use of cache.

As a rule, always ask yourself how frequently a piece of data is needed.

The more times it is needed, the more effective the use of cache will be.

To answer this question, you need to have a really good understanding of the statistical distribution of data access from your data source.

Caching is more likely to be effective for your use case when your data has a normal bell-curve distribution instead of flat distribution in terms of access.

5 - Does the original operation have side effects?

If you want to cache the result of an operation, it must not have side effects. For example, the operation should not store data, make changes to other systems or control some software or hardware item.

If the result of such operations is cached, you will end up breaking your application when the requests are served from the cache and the side effects are ignored.

That’s it

The decision to use a cache is not an emotional matter.

Though caching has a very important role in application development, you need to run your use case through the lens of hard questions.

If you find satisfactory answers that point to the use of a cache, you will be able to reap the benefits in the long run. Otherwise, caching systems can become more of a liability.

Do you want to stay updated and relevant to the latest trends in technology?

Do you also want to get actionable advice about web development, cloud computing, distributed systems, platform engineering and software architecture?

If yes, subscribe to the ProgressiveCoder newsletter for FREE.

5 Must-Avoid Pitfalls in Software Development

Saurabh Dashora — Thu, 02 Feb 2023 13:05:07 +0000

Software development is the ultimate test of patience, problem-solving skills and sheer determination.

But even the most seasoned developers can fall into common pitfalls that throw a wrench into their coding magic.

Yes, I'm talking about those things that seem like a great idea at the time but end up causing chaos and frustration later on.

In this post, I'm going to take you through 5 software development pitfalls that you must avoid like the plague.

So put on your best 'I won't make these mistakes' grin, and let's get started!

Pitfall#1 - Not working on clearly defined requirements

Picture this - You've spent countless hours pouring your heart and soul into a project, only to realize that it's not quite what your client had in mind.

Maybe, the client didn't define their requirements and you didn't bother to make them do so.

The problem is that without clear and specific requirements, you're basically coding in the dark.

And let's be honest, who wants to waste their time building something that nobody wants?

How do you avoid this pitfall?

Here are a few tried and tested ways:

Conduct stakeholder interviews
Create user stories
Clarify vague or conflicting requirements

Pitfall#2 - Underestimating the Time and Effort Required

This one's my favourite. I have seen it happen so many times that if I give myself a dollar for every time I saw it, I'd be a millionaire by now.

It's the good old game of 'How quickly can I get this done?'

For some reason, software developers are guilty folks. When someone asks them to provide an estimate, they always think that they are doing something wrong by giving a reasonable number.

I really wish developers would cut themselves some slack.

Creating software is like a puzzle with a million pieces. And if you don't account for each and every one of those pieces and then some more, things can get messy pretty fast.

Here are some things you can do to avoid falling into the trap of missed deadlines and reduced quality:

Be realistic about the time and effort required for a particular requirement.
Use project management tools to keep yourself in check.
Regularly revisit your project timeline to make sure you stay on track.

Pitfall#3 - Neglecting User Feedback

User feedback is like the voice of the people and the pulse of the product.

Don't be that developer who shuts their ears to the words of the user.

Of course, it does not mean that you agree to all their demands even if they are impossible. But your attempt should be to collaborate with them to deliver the best possible product.

When in doubt, remember that software development is all about solving problems for real people. If you neglect the feedback of the user, you risk everything.

I suggest that you should always try to understand the user's viewpoint before starting to write code. That will tell you the WHY of the product.

Conduct user testing, show prototypes and incorporate feedback into your development plans. Take more time if needed. Trust me, your users will thank you.

Pitfall#4 - Not Doing Proper Testing and Debugging

"We will figure it out later".

This is the single biggest lie in software development. Your job as a software developer is to figure things out before they happen.

Just like a tightrope walker needs a net, your code needs proper testing and debugging to keep it from crashing down.

Neglecting these steps can lead to all sorts of issues like slow performance and a string of bugs.

While building your application, dedicate time for both automated testing and manual testing along with debugging.

Don't be in a rush to deliver software before all the major issues are fixed.

Pitfall#5 - Neglecting Security

Don't be that guy! Trust me, there's no fun in that.

Yes, I'm talking about the developer that ignores security.

As a developer, it's your job to conduct regular security audits, use secure coding practices and stay up-to-date with the latest security trends.

However, even if you are not experienced enough in complicated security matters, you should not feign ignorance.

Try to find someone who knows about it. Ask around and take help. Get your application reviewed by the security department if there's one. Delay the project if you must but don't take security matters for granted, or you'll be sorry!

That's it for now!

These are the 5 must-avoid pitfalls that I could think of right now.

But there may be more. Don't hesitate to write about them in the comments section below. After all, software developers should have each other's back.

If you found this post useful, do share it with your friends and colleagues.

You can also connect with me on other platforms:

My Regular Newsletter

Twitter

Youtube

9 Must-Know Rules for Writing Clean Code

Saurabh Dashora — Mon, 30 Jan 2023 09:40:35 +0000

Are you tired of staring at a mess of code that even your cat wouldn’t touch?

Look no further. In this post, I will tell you 9 rules to turn your spaghetti code into a work of art.

So grab a cup of coffee, sit back and take charge of your code!

Rule#1 - Keep your code organized and easy to read

Don’t organize your wardrobe but definitely organize your code. By keeping your code organized and easy to read, you can find what you need in a snap.

Check the below example:

// Group related functions together
function calculateTotal() {
  // code
}

function calculateTax() {
  // code
}

function calculateDiscount() {
  // code
}

// Use comments to explain the purpose of the code
function calculateTotal() {
  // calculate subtotal
  let subtotal = 0;
  // add items
  subtotal += item1.price;
  subtotal += item2.price;
  // return total
  return subtotal;
}

As you can see, a little bit of planning and a healthy dose of comments can make it a breeze to navigate your code.

At one glance, you get a good idea of the overall functionality even after it’s been a while since you worked on the code.

It’s not every day that you thank your past self for doing something good!

Rule#2 - Follow Established Coding Conventions and Standards

I know it is tempting to be a rebel and chart your own course.

But sometimes, it pays to follow established coding conventions and standards. This includes things like naming conventions, commenting guidelines and more.

See the below snippet for what I’m trying to say.

// Naming conventions
let userName; // camelCase for variables and properties
const MAX_USERS; // UpperCamelCase for constants

// Commenting guidelines
// This function takes in a number and returns its square
function square(num) { 
  return num * num;
}

The idea is to play the game with consistency. By following the code conventions of your team, your code won’t stick out like a sore thumb and your fellow developers will appreciate you.

Rule#3 - Use descriptive variable and function names

I know it’s stylish to name your variables X and Y.

But trust me, even you won’t be able to understand their true meaning after a few months.

It’s always safe to give descriptive names to variables. It might even be better for you in the long run.

// Descriptive function name
function calculateSum(numbers) {
  // Descriptive variable name
  let sum = 0;
  // Iterate through the array of numbers
  for (let i = 0; i < numbers.length; i++) {
    // Add each number to the sum
    sum += numbers[i];
  }
  // Return the sum
  return sum;
}

As you can see, even if the function’s code has a loop, the function and variable name makes the objective of the program absolutely clear.

Rule#4 - Avoid Hard-Coding Values and Use Constants

Say goodbye to hard coding and hello to constants!

By using constants, you’ll be able to store values that you know won’t change throughout your program. This will help you avoid the headache of having to search and replace values throughout your code.

Check out the below code.

const TAX_RATE = 0.07;

function calculateTotal(subtotal) {
  return subtotal + (subtotal * TAX_RATE);
}

If the tax rate changes in the future, you can easily update the value in one single place and have it reflected throughout your code.

Rule#5 - Keep functions small and focused

When you are starting off as a developer, it’s always tempting to turn your functions into swiss knives. The feeling of explaining that your code can do so much stuff is so enticing.

As you become a seasoned developer, this habit wears off. But sometimes it doesn’t.

Trust me on this. Keep your functions small and your life will be happy.

If you write small and focused functions that are easy to understand, chances are that you won’t be disturbed on your beach vacation if something goes wrong.

See this example.

function calculateTotal(subtotal, tax, discount) {
  const total = subtotal + tax - discount;
  return total;
}

The above function is short and to the point. This makes it easy to understand and maintain.

Rule#6 - Use Appropriate Data Structures

Are you always trying to fit a square peg in a round hole? If yes, it’s time to use appropriate data structures.

Just like a carpenter has a variety of tools for different tasks, a developer should have a variety of data structures for different types of functionality.

Here’s a cheat sheet:

Use arrays when you need to store a collection of items that have a specific order.
Use lists when you need to store a collection of items that can change dynamically.
Lastly, use maps if you need to store a collection of items that can be accessed by a key.

Check out the below code that demonstrates the use of different data structures.

// Using an array to store a collection of items that have a specific order
const shoppingList = ["milk", "bread", "eggs"];
console.log(shoppingList[1]); // Output: "bread"

// Using a list to store a collection of items that can change dynamically
const todoList = ["write code", "debug", "test"];
todoList.push("deploy");
console.log(todoList); // Output: ["write code", "debug", "test", "deploy"]

// Using a dictionary to store a collection of items that can be accessed by a key
const phonebook = {
  "John": "555-555-5555",
  "Mary": "555-555-5556",
  "Bob": "555-555-5557"
};
console.log(phonebook["Mary"]); // Output: "555-555-5556"

By using an appropriate data structure based on the requirement, you’ll find that your code is not only efficient but also easy to understand.

Rule#7 - Use Version Control

Just like your application is useless if it only runs on your machine, your code is useless if it isn’t committed to a central repository.

Every developer should get used to version control. Don’t forget to commit your code regularly. If you are doing it, make sure others on your team also do it.

All it takes is a few commands:

// Initialize a new Git repository
$ git init

// Add all files to the repository
$ git add .

// Commit the changes with a message
$ git commit -m "Initial commit"

A good version control tool allows developers to track changes, collaborate with others and easily revert to previous versions in case of any issues.

Rule# 8 - Automate Repetitive Tasks

Don’t be like a hamster on a wheel, constantly running in circles and doing the same boring tasks over and over again.

You should use tools and scripts to automate repetitive tasks in your code. This will not only save time but also make your code more reliable and efficient.

Check out this code example of a simple testing automation script.

const testCases = [
    { input: [1, 2], expectedOutput: 3 },
    { input: [2, 3], expectedOutput: 5 },
    { input: [3, 4], expectedOutput: 7 },
];

testCases.forEach(testCase => {
    const output = addNumbers(testCase.input[0], testCase.input[1]);
    if (output !== testCase.expectedOutput) {
        console.error(`Test failed: expected ${testCase.expectedOutput} but got ${output}`);
    } else {
        console.log("Test passed");
    }
});

You can also automate the building process for compiling your code and creating a final package.

See the below example where we automate a build process using a tool like Gulp.

// Automating a build process using a tool like Grunt or Gulp
const gulp = require('gulp');

gulp.task('build', function() {
// Build process logic
});

gulp.task('default', ['build']);

Rule#9 - Keep your code up-to-date

Don’t be a dinosaur. You’re only going to go extinct.

Keep your code up-to-date. Update your application dependencies whenever possible.

For example, if you’re working in the Node ecosystem, keep the NPM up-to-date and upgrade your development environment.

// Keep dependencies up-to-date using package manager
$ npm update

// Keep the development environment up-to-date
$ nvm install 12.16.3

That’s it!

Well, there you have it. 9 rules to help you write clean and efficient code.

Of course, these are not the only things that matter. But by following them, you’ll be able to start on the path to writing code that not only works well but also appears pleasing for others to read, understand and maintain.

And if you have any other rules, do mention them in the comments section below.

You can also connect with me on other platforms:

My Regular Newsletter

Twitter

Youtube

Build a Node.js API Proxy to supercharge your backend

Saurabh Dashora — Thu, 26 Jan 2023 09:28:20 +0000

Have you ever wondered who's the middleman between you and that fancy API you're using?

Meet the API proxy. The undercover agent of the tech landscape!

Just like how your best friend proxied your attendance call during that boring college lecture, an API proxy acts on behalf of something else in the digital realm.

And it's not just for the FBI. Even API developers use it to sneakily redirect incoming requests to the real server.

Want to learn how to build a Node.js API Proxy on your own? Keep reading!

The Need for an API Proxy

Several reasons justify the use of an API proxy:

Maybe the real API server is unpredictable. A proxy can act as a trusty stable hand and provide a more stable interface for your clients.
Probably the current API server is like a temporary rental car. You have plans for upgrading in the future, but you want to shield the API clients from future changes. You can use an API proxy to make the transition smoother.
Or maybe, the response from the API server is like a bad translation. A proxy can act as a translator and convert the API response to something that the clients can understand or consume.

All in all, an API Proxy makes your actual backend API more scalable in the long run.

Bottom line, an API Proxy is like a swiss-army knife. You never know when you might need one. But when you do, you would be happy to have it.

The Project Setup

First, you need to initialize the project by executing the below command in a project directory:

$ npm init -y

This will generate a basic package.json file with meta-data information about the project such as name, version, author and scripts.

Next, install a couple of packages for developing the Node.js API proxy.

$ npm install --save express http-proxy-middleware

express is a minimalistic web framework you can use to build API endpoints.
http-proxy-middleware is a third-party Node.js package that has all the tools to create an API proxy

After the package installation, define a start command for the project within the package.json file. You can use this command to start the application.

Your project’s package.json should look similar to the below example.

{
  "name": "express-proxy-demo",
  "version": "1.0.0",
  "description": "Demo Application for Proxy Implementation in Node.js",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "author": "Saurabh Dashora",
  "license": "ISC",
  "dependencies": {
    "express": "^4.18.2",
    "http-proxy-middleware": "^2.0.6"
  }
}

Creating the API Proxy using Node.js

The project setup is done. Now, it’s time to build the API Proxy.

You are going to create an application that acts like a proxy and redirects incoming requests to an API that’s hosted somewhere else. Think of it like a game of telephone but with requests.

For the sake of this tutorial, I’m going to use some fake APIs hosted at the wonderful website JSONPlaceholder.

Check out the below illustration for a visual perspective.

Here’s the index.js file where all the magic happens.

const express = require('express');
const { createProxyMiddleware } = require('http-proxy-middleware');

const app = express();

const PORT = 3000;
const HOST = "localhost";

const API_URL = "https://jsonplaceholder.typicode.com";

app.get("/status", (req, res, next) => {
    res.send('This is a proxy service');
});

const proxyOptions = {
    target: API_URL,
    changeOrigin: true,
    pathRewrite: {
        [`^/api/posts`]: '/posts',
    },
}

const proxy = createProxyMiddleware(proxyOptions);

app.use('/api/posts', proxy)

app.listen(PORT, HOST, () => {
    console.log(`Proxy Started at ${HOST}:${PORT}`)
});

Let’s understand each step in the above program:

Step 1: The first segment of the code contains the import statements for express and http-proxy-middleware.

Step 2: The next statement creates an application instance using the call to express() function. This is followed by declaring a few important constants such as PORT, HOST and API_URL.

Step 3: Implement an endpoint /status to describe the role of the application. This endpoint has nothing to do with proxying requests. It simply provides a way to test our application.

Step 4: Next, we declare an object proxyOptions. This is a configuration object for our API proxy. It contains a few important properties

target - It defines the target host where you want to proxy requests. In our case, this is the https://jsonplaceholder.typicode.com
changeOrigin - This is set to true since we are proxying to a different origin.
pathRewrite - This is a very important property where you define the rules for rewriting the path. For example, the expression [^/api/posts]: '/posts' routes all incoming requests directed at URL /api/posts to just /posts. In other words, this will remove the /api prefix from the path.

Step 5: After declaring the configuration object, create the proxy object by calling createProxyMiddleware() function with the proxyOptions object as input.

Step 6: Next, create a request handler for the path /api/posts and pass the proxy object as handler for the incoming request.

Step 7: At the very end, start the Node.js API Proxy server to listen on the port and host that were already declared earlier.

You can start the application using the command npm run start.

> express-proxy-demo@1.0.0 start
> node index.js

[HPM] Proxy created: /  -> <https://jsonplaceholder.typicode.com>
[HPM] Proxy rewrite rule created: "^/api/posts" ~> "/posts"
Proxy Started at localhost:3000
Messages about the proxy setup indicate that the proxy is configured properly.

If you visit the URL http://localhost:3000/api/posts/1 in the browser, you will get the response from the JSONPlaceholder APIs as below:

{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\\nsuscipit recusandae consequuntur expedita et cum\\nreprehenderit molestiae ut ut quas totam\\nnostrum rerum est autem sunt rem eveniet architecto"
}

Basically, the Node.js API Proxy is doing its job by proxying requests to the mock APIs hosted by JSONPlaceholder.

That’s it

Congratulations! You've just built a basic version of a Node.js API proxy.

But don't stop there. Just like a birdhouse, you can always make it bigger and better based on specific requirements.

The http-proxy-middleware is like the toolbox for building a Node.js API proxy server and you can extend it in many ways using path matching and so on.

The code for this demo is available on GitHub.

If you found the post useful, consider sharing it with friends and colleagues. And if you have any questions, feel free to ask in the comments section below.

In case you want to know more about proxies, you can also head over to this tutorial on building a Node.js forward proxy.

You can also connect with me on other platforms:

My Regular Newsletter

Twitter

Youtube

Don't Unleash Zalgo in your Node.js Application!

Saurabh Dashora — Thu, 12 Jan 2023 11:57:12 +0000

Don't Unleash Zalgo in your Node.js application.

That's my sincere advice.

And I believe that you can easily avoid the traps of Zalgo while building your next API. But for that, you need to understand how it looks like.

In the real world, the term Zalgo comes from a meme. It happens to be an Internet legend about an ominous entity believed to cause insanity, death and destruction of the world. Zalgo is often associated with scrambled text on web pages and photos of people whose eyes and mouths have been covered in black.

In the matrix-world of software development, Zalgo is a piece of code that can destroy your application.

And you won't even know what's going on.

The First Signs of Zalgo - Confused Code

Callbacks make it possible for JavaScript to handle concurrency despite being single-threaded (technically).

This is what essentially drives concurrency in Node.js.

You can build callbacks that behave synchronously or asynchronously. Typically, you choose only one approach.

The problem shows up when you end up choosing both - sync and async.

In other words, what if you develop a function that runs synchronously in certain conditions and asynchronously in some other conditions?

It's an unpredictable function.

Unpredictable functions and the APIs built using them unleash Zalgo in your code.

Example

Here's an example:

let cache = {};

function getStringLength(text, callback) {
    if (cache[text]) {
        callback(cache[text])
    } else {
        setTimeout(function() {
            cache[text] = text.length;
            callback(text.length)
        }, 1000)
    }
}

Despite the innocent appearance, the getStringLength() function is evil.

Though it is used to simply calculate the length of a string, it has two faces.

If the string and its length are available in the cache object, the function behaves synchronously by returning the data right away from the cache.

Otherwise, it calculates the length of the string and stores the result in the cache before triggering the callback. The calculation happens asynchronously using a setTimeout().

Do note that the use of setTimeout() is to force an asynchronous behaviour. You can replace it with any other asynchronous activity such as reading a file or making an API call. The idea is to demonstrate that a function can have different behaviour in different situations.

“But how does it unleash Zalgo?” you may ask.

Let us write some more logic to actually use this unpredictable function. Check it out below:

function sleep(milliseconds) {  
      return new Promise(resolve => setTimeout(resolve, milliseconds));  
}  

let cache = {};

function getStringLength(text, callback) {

    if (cache[text]) {
        callback(cache[text])
    } else {
        setTimeout(function() {
            cache[text] = text.length;
            callback(text.length)
        }, 1000)
    }
}

function determineStringLength(text) {
    let listeners = []
    getStringLength(text, function(value) {
        listeners.forEach(function(listener) {
            listener(value)
        })
    })
    return {
        onDataReady: function(listener) {
            listeners.push(listener)
        }
    }
}

async function testLogic() {
    let text1 = determineStringLength("hello");
    text1.onDataReady(function(data) {
        console.log("Text1 Length of string: " + data)
    })

    await sleep(2000); 

    let text2 = determineStringLength("hello");
    text2.onDataReady(function(data) {
        console.log("Text2 Length of string: " + data)
    })
}

testLogic();

Pay special attention to the determineStringLength() function. It is a sort of wrapper around the getStringLength() function.

The determineStringLength() function creates a new object that acts as a notifier for the string length calculation. When the string length is calculated by the getStringLength() function, the listener functions registered within determineStringLength() get invoked.

To test this concept, you have the testLogic() function at the very end end.

The test function calls determineStringLength() function twice for the same input string “hello”. Between the two calls, we pause the execution for a couple of seconds using the sleep() function. This is just to introduce a bit of time lag between the two calls.

Running the program provides the below result:

Text1 Length of string: 5

Only one statement is printed. The callback for the second operation never got invoked.

For text1, the getStringLength() function behaves asynchronously since the data is not available in the cache. Therefore, the listener got registered and the output was printed.
For text2, none of this happens. It gets created in an event loop cycle that already has the data available in the cache. Therefore, getStringLength() behaves synchronously and the callback that's passed to it gets called immediately. This in turn calls all the registered listeners synchronously. However, registration of new listener happens later and hence it is never invoked.

The root of this problem is the unpredictable nature of getStringLength() function. Instead of providing consistency, it increases the unpredictability of our program.

Such bugs can turn out to be extremely complicated to identify and reproduce in a real application. More often, they cause nasty bugs and unleash Zalgo in our application.

So, how can we avoid all of this mess?

Use Deferred Execution

It might appear tricky but preventing such situations can be quite simple. Just make sure the functions you are writing behave consistently in terms of synchronous vs asynchronous behaviour.

For example, check the below code:

function getStringLength(text, callback) {

    if (cache[text]) {
        process.nextTick(function() {
         callback(cache[text]);
        });
        //callback(cache[text])
    } else {
        setTimeout(function() {
            cache[text] = text.length;
            callback(text.length)
        }, 1000)
    }
}

Instead of directly triggering the callback, you can wrap it inside the process.nextTick(). This defers the execution of the function until the beginning of the next event loop phase.

Subtle reasons can cause nasty bugs. Unleashing Zalgo is one of them and hence, an interesting name was given to this situation.

As I mentioned in the beginning, the term was first used by Isaac Z. Schlueter who was also inspired by a post on Havoc’s Blog. Below are the links to those blogs:

Designing APIs for Asynchrony
Callbacks, synchronous and asynchronous

You can check out those posts to get more background. I hope the example in this post was useful in understanding the issue on a more practical level.

If you liked this post, consider sharing it with friends and colleagues. You can also connect with me on other platforms.

My Regular Newsletter

Twitter

Youtube

Building a Node.js Webserver with HTTP Module is super easy!

Saurabh Dashora — Thu, 05 Jan 2023 06:13:36 +0000

What if your car breaks down on a highway and you have no clue what the mechanic is saying about the problem?

This is the same as working with your Node.js webserver using frameworks such as Express. You know how to build applications. But you don't know how those applications work.

And that's usually fine until you have a need to leverage the platform more efficiently.

There are many easy ways to build a webserver by using frameworks such as Express. However, it is also not that hard to build a basic Node.js Webserver with the core HTTP module and nothing else.

Even if you might not use it to build real applications, knowing how it's done has benefits if you want to become a real Node.js practitioner. Just like being an automobile enthusiast instead of a simple driver.

Surely, you don't want to be just a driver!

All right - so now that you are hopefully convinced, let's build an extremely simple webserver in Node.js.

Node.js Basic Webserver

Create a file named index.mjs in a project directory and paste the below code into the file.

import { createServer } from "http";

const server = createServer();

server.listen(8080, () => {
    console.log(`Server is listening to http://localhost:${server.address().port}`);
});

And that's all the code that you need to create a Node.js webserver.

But what are you doing over here?

First, you import createServer() function from the http module. This is a core module that ships with the Node.js installation.
Next, you call the imported createServer() function to get a server object. You store the object in a constant named server.
The server object has a method named listen() that takes a couple of arguments. One is the port number and the second is a callback function that prints a message when the server is ready to handle requests.

To start the server, you can execute the command node index.mjs.

Of course, you can say that this server is practically useless. And you won't be exaggerating.

When you visit http://localhost:8080, nothing happens. If you are lucky, you might get a timeout.

Even though the server is up and running, it does not know what to do with an incoming request.

Handling HTTP requests

So, how can you make the Node.js webserver somewhat useful?

You need to give it the ability to handle requests.

Check out the below code:

import { createServer } from "http";

const server = createServer((request, response) => {
    response.writeHead(200, {'content-type': 'text/plain; charset=utf-8' })
    response.write('Hello');
    response.end(' World\n');
});

server.listen(8080, () => {
    console.log(`Server is listening to http://localhost:${server.address().port}`);
});

The createServer() function now takes a callback with two input objects - request and response. The job of this callback is to handle every incoming request.

The response object helps us manipulate the output. You can use the writeHead() method to set the response status code and response headers. For example, you can set the content-type to text/plain and charset to utf-8.

To set the actual response, you can use the response.write() method to set the response string. You can also have multiple response.write() statements one after the other.

Once the response is written, you need to tell the webserver to close the stream. This is done using the response.end() method. The response.end() method can also take some part of the response.

But who uses a webserver to send some text response?

Of course, there might be a requirement but it does not seem like the best use of a webserver.

How about sending some HTML as a response?

Sending HTML Response

Check out the below code:

import { createServer } from "http";

const server = createServer((request, response) => {
    response.writeHead(200, {'content-type': 'text/html; charset=utf-8' })

    const body = `<!DOCTYPE html>
        <html>
            <head>
                <meta charset="utf-8">
                <title>Node.js Webserver Demo</title>
            </head>
            <body>
                <h1 style="color: orange">Hello World</h1>
            </body>
        </html>`;

    response.end(body);
});

server.listen(8080, () => {
    console.log(`Server is listening to http://localhost:${server.address().port}`);
});

Despite the size of the code, there are only minor modifications.

One is setting the content-type value to text/html. The second is declaring a constant named body with some HTML content and sending it as part of response.end() method.

If you access http://localhost:8080 now, you should see the HTML page displayed in the browser.

At this point, you might feel somewhat satisfied with the capabilities of the Node.js webserver. But the nature of the content is still pretty static.

Typical web applications consist of dynamic web pages. Output depends on the request made by the user.

How can you make your Node.js webserver generate dynamic responses?

Sending Dynamic Response

The changes are pretty straightforward. See below:

import { createServer } from "http";

const server = createServer((request, response) => {
    response.writeHead(200, {'content-type': 'text/html; charset=utf-8' })

    const url = new URL(request.url, 'http://localhost:8080');

    const body = `<!DOCTYPE html>
        <html>
            <head>
                <meta charset="utf-8">
                <title>Node.js Webserver Demo</title>
            </head>
            <body>
                <h1 style="color: orange">Hello ${url.searchParams.get('name')}</h1>
            </body>
        </html>`;

    response.end(body);
});
server.listen(8080, () => {
    console.log(`Server is listening to http://localhost:${server.address().port}`);
});

The major change in the above snippet is that you are now reading the URL requested by the client using the request.url property.

For example, if the user enters the URL http://localhost:8080/?name=adamsmith in the address bar of the browser, the request.url will contain the value /?name=adamsmith.

To display the name in the HTML response, I recommend using the WHATWG URL API to create a URL instance. The URL API helps you parse URLs and break them into individual components. It will produce an object that represents the requested URL with all of the components.

Within the HTML string, you can extract the query parameter name from the URL object's searchParams property using the expression url.searchParams.get('name').

If you access http://localhost:8080/?name=adamsmith after making the above change, you will see Hello adamsmith displayed in the browser window.

This is a dynamic HTML output that depends on what you pass in the request's query parameters.

There is no doubt using a framework makes life easy for developers. You ultimately write less code and focus more on business logic. Also, you write less boilerplate code to handle basic things when using a framework.

But knowing how things work under the hood of any framework is also important. Learning how to create a Node.js webserver with just the core HTTP module gives you important lessons on how the frameworks work internally.

If you enjoyed this post or found it helpful, do share it with friends and colleagues. Also, here are some other articles you might find useful:

How does Node.js Reactor Pattern work?

Various Phases of the Node.js Event Loop

You can also connect with me on other platforms:

My Regular Newsletter

Twitter

Youtube

I misunderstood Scalability in a Distributed System

Saurabh Dashora — Wed, 28 Dec 2022 08:53:48 +0000

"Is the application scalable?" the manager asked. It was almost halfway through the weekly system roadmap meeting. I thought we were done with all the tech parts and would soon move to business-related questions.

The architect of my team cleared his throat and answered, "It can easily handle double the usual load."

The manager gave a long look as if he didn't believe the answer. But, in the absence of any better points, he continued with further updates. I implicitly believed that our system is indeed scalable. After all, the architect had announced it in clear terms.

A year down the line, I learned the truth. One fine day, the system crashed due to excessive load.

The bank had launched a new loan product for the holiday season. The business strategy paid off and there was a surge in demand. Since our application provided the core data to make the credit decision, it was swamped with 5 times more incoming requests from several other applications.

We added more CPUs to handle the load on our service. The same strategy had worked the last couple of times. However, this time it didn't.

Our system was scalable until it no longer was.

Upon analysis, we found that the issue wasn't entirely in our application code. Our application was made up of multiple services that connected with the bank's central database. The connection to the database was made using a connection pool.

Our fault was that we were careless about the number of open connections. Turns out, the database administrations were not.

The DBAs had a hard block on the number of open connections for our group of applications. When the incoming requests to our application surged, we ended up acquiring even more connections until we hit the wall. No matter how many additional CPUs we added to increase the application capacity, it didn't make any difference.

To add insult to injury, the DBAs refused to increase the permissible number of connections. The only option left was a costly change to our application to handle connections properly. For all practical purposes, our system was no longer scalable.

Of course, it was our mistake. We should have accounted for the limited number of connections and used them wisely. Nevertheless, the problem was not unfixable. But the change wasn't cost-effective.

Though the situation was ultimately handled, it piqued my interest in the subject. We thought the system was scalable. But it wasn't. At least not until we hit the connection limit.

It was clear we were wrong about the true meaning of scalability. I decided to dig deeper into the whole situation. My initial motivation was to avoid getting into the same situation as my team's architect

The Usual Definition of System Scalability

While studying scalability for distributed systems, the most common definition I have come across is as follows:

Scalability is the ability of the system to handle an increased workload.

How does this definition apply in real life?

Developers of an actively used system monitor its workload levels. They create processes for predicting when the performance of the system will become unsatisfactory. The goal is to increase the system's capacity before the danger level is reached.

In this approach to scalability, the primary focus is to determine an interval of demand within which the system will perform at an acceptable level. If this interval is sufficiently large, the system is considered scalable.

In my view, this definition is pretty limited. The focus of this definition is always on the workload.

The main concern of this definition is that a system is scalable if it continues to perform adequately as the workload grows. No attention is given to how a system will have to be modified to keep the performance at the same level.

As we found in our case, our system could have continued performing at a good level. But the changes required on the database side were not acceptable. Neither was changing the code cost-effective.

The common scalability definition pays no attention to the approach of capacity increase or its overall impact on the system. We don't ask important questions such as:

Will doubling the number of processors allow the system to handle double the workload?
Is the overhead of coordinating work between processors going to be prohibitively high?

In defence of the definition, it is the most common definition used by system designers and developers. We hear it all the time when someone says that a system is scalable and it can handle double the workload with no problems.

Scalability by Extension

In my quest to understand scalability, I came across another definition of scalability that often gets neglected. However, I have come to believe that it may be far more important in practical situations.

Scalability is the ability to handle an increased workload by repeatedly applying a cost-effective strategy for extending a system's capacity.

Straight away, the focus shifts towards the strategy for adding capacity.

We are no longer interested in a one-time capacity increase to improve scalability. We are interested in a strategy for adding capacity and the number of times the strategy can be applied cost-effectively.

This line of thought always makes me think about world-class tennis players like Rafael Nadal or Roger Federer.

Over the years, how many times have players like Nadal or Federer adapted their playing style to the changing demands of the game?

Many times, I reckon.

Each change they made prolonged their careers and made them more successful.

However, making a change is not easy. Many players rise to the top and fizzle out because they could not adapt. Even the best players are bound to face difficulties in trying to adapt. The easier it is for a player to make changes, the higher the probability for that player to maintain his dominance in the game.

The same goes for scaling a system.

When we consider the second definition of scalability, we start considering other types of arguments about our system.

If we add more processors to increase capacity, what is the method for coordinating work between the added processors?

Will the coordination method take more processing cycles?

If yes, the full benefit of adding capacity won't be realized. Hence, adding processors beyond a certain point may not be a cost-effective approach to improving scalability.

Focusing on the repeated application of a scaling strategy makes us more cognitive of our choices.

For example, replacing an O(n^2) with an O(nlogn) algorithm makes it possible to process a larger workload in the same amount of time. In other words, replacing an algorithm with a more efficient one improves the system's scalability.

But can we use this approach repeatedly?

I don't think so.

Once we have the most efficient algorithm in place, the algorithmic replacement strategy stops being viable. You cannot keep applying the strategy again and again to improve the system's scalability.

Is a System Scalable?

Armed with the two definitions of scalability, I was finally able to make sense of the fundamental question. This was the same question that popped up in our meeting and was answered inadequately.

Is a system scalable?

To answer the question, we end up labelling a system as scalable or not. In my view, this is an oversimplification.

What does it mean for a system to be scalable?

Most systems scale in some sense. However, no system is infinitely scalable.

So how should we approach scalability?

Theoretical Approach

Rather than labelling systems, it is more fruitful to compare the scalability of two different systems. Consider the below graph that shows the response vs demand curves for two hypothetical systems A and B.

For any given demand level, the response is worse for System A as compared to System B. If there is a maximum tolerable value for the response, System A will reach it earlier than System B.

System B is more scalable than System A.

Of course, if both lines continue to rise at the same monotonic rate, they will eventually reach a point where the demand for resources exceeds their availability. At that point, the response times will become unsatisfactory for both systems. Though the points may be different for A and B, they signify the limit of a system's scalability.

Remember - no system is infinitely scalable.

Systems do not follow a monotonic rate of increase in response vs demand metric. The curve looks more like the below example.

The hypothetical system tolerates demand increase quite well until it hits a significant design limitation. This limitation forms a knee-like shape in the response demand curve. After crossing a particular demand level, the response metric simply goes out of hand.

The goal of designers is to keep the knee of the curve as far to the right as possible. Once a system reaches closer to the knee, it is no longer scalable. Any increase in demand will push it down the hill.

Practical Approach

The theoretical aspects of scalability were genuine eye-openers for me. But I still thought something was missing.

I couldn't completely relate the above graphs to our system. We were able to scale our system successfully during the initial days whenever there was a general increase in load. Things went south only when we hit the database limit.

This behaviour does not fit into either of the two theoretical models to depict scalability.

How to classify this behaviour?

Most real system exhibit a more hybrid behaviour. The below graph shows this:

When a system is running in the bottom green zone, it is responding well. As it begins to linger in the middle zone (shown in yellow), the response starts to get unacceptable. When it finally crosses into the top zone (shown in red), the system becomes unusable.

In my opinion, this is a more accurate representation of the second definition of scalability.

On the first couple of occasions, the system designers were able to take quick action to keep the response within acceptable limits. It was probably a matter of adding additional memory.
On the third occasion, the designers needed more time and effort to bring the response metric down to acceptable levels. The changes needed to improve the response were probably more complicated.
In the last case, the system reached a point where no reasonable solution was possible to keep the response times from becoming unusable. The system was no longer scalable using the cost-effective techniques that worked the initial few times.

The above turn of events also depicts the situation of our system. Even though we were able to keep our system scalable for some time, it ultimately reached a point beyond which any cost-effective solution was not possible.

So, is the system scalable?

The answer is - it depends! And not in a cop-out sort of way.

Scalability is more of a moving target that evolves rather than a fixed state.

If the system owner can afford to continue pouring money to meet the higher demand levels, the system is scalable up to a certain point. Beyond this point, no cost-effective action can alleviate the response metric issues.

If the system owner has no money to purchase additional resources at the first very hiccup, the system is not scalable.

Whatever the case, no system is infinitely scalable.

Concluding Thoughts

My journey to understand the true meaning of scalability led to some important conclusions.

In my view, you cannot simply slap a label of scalability on a system and call it a day.

All systems are scalable to some extent. But it does not mean that you can continue scaling them indefinitely in a cost-effective manner.

An ideal scalability situation requires a nuanced approach to system design. Only focusing on adding resources to improve scalability is a trap. You also need to consider the cost-effectiveness of the additional resource.

With this post, my goal was to bring this perspective to the discussion.

Do share your thoughts in the comments section below.

If you enjoyed this article or found it helpful, consider sharing it with your friends and colleagues.

You can also connect with me on other platforms:

Twitter

Youtube

How to use Svelte Keyed Each Block?

Saurabh Dashora — Tue, 18 Jan 2022 12:28:08 +0000

In this post, we learn how to use Svelte Keyed Each Block.

In the normal Svelte Each Block, we use the each keyword to render an array or list. While it works perfectly when we have static lists, there is a serious issue when we delete items from the list. We are going to first look at the issue and then come up with the solution to it.

1 – Svelte Each Block Issue

To demonstrate the issue, we will take the example of displaying books and their respective authors.

Below is our App component.

`App.svelte`

<script>
    import Book from "./Book.svelte"

    let books = [{
        bookName: "Eye of the World",
    },
    {
        bookName: "The Way of Kings",
    },
    {
        bookName: "The Name of the Wind",
    }]

    function handleClick() {
        books = books.slice(1);
    }

</script>

<h1>Welcome to the Fantasy Library</h1>
{#each books as book, idx}
<h3>{idx + 1}</h3>
<Book 
    bookName = {book.bookName}
/>
{/each}
<button on:click={handleClick}>
    Remove first Book
</button>

In the above piece of code, we have the books array with three hard-coded books. We use the each block to render the books using the Book component. Finally, we also have a button to remove the first book from the each block. When the user clicks the button, we invoke the handleClick() function that slices the books array.

Below is the code for the Book component.

`Book.svelte`

<script>
    const authors = {
        "Eye of the World": "Robert Jordan",
        "The Way of Kings": "Brandon Sanderson",
        "The Name of the Wind": "Patrick Rothfuss",
    }

    export let bookName;

    const author = authors[bookName];
</script>
<div>
    <span>Book Name: {bookName} // Author: {author}</span>
</div>

In this component, we have the book to author map. Based on the input bookName, we basically determine the name of the author.

If we run the app now and then click the button to remove first book once, we will see the below output.

As you can see, the bookName and the authorName does not match. Though the first book was correctly deleted, the authorName got mixed up.

Why does this happen?

The reason is that when we modify the underlying array of an each block, Svelte adds and removes items from the end of the block. It also updates any values that have changed. This is the default behaviour.

Due to this, the last item is removed and the items above it are updated. However, the author value is not updated because it is the local constant of the Book component. Its value was fixed at the time of component initialization. This leads to the data mismatch.

2 – The Svelte Keyed Each Block

To get around this issue, we need to use keyed each block.

See below example:

`App.svelte`

<script>
    import Book from "./Book.svelte"

    let books = [{
        id: 1,
        bookName: "Eye of the World",
    },
    {
        id: 2,
        bookName: "The Way of Kings",
    },
    {
        id: 3,
        bookName: "The Name of the Wind",
    }]

    function handleClick() {
    books = books.slice(1);
    }

</script>

<h1>Welcome to the Fantasy Library</h1>
{#each books as book, idx (book.id)}
<h3>{idx + 1}</h3>
<Book 
    bookName = {book.bookName}
/>
{/each}
<button on:click={handleClick}>
    Remove first Book
</button>

Basically, here we introduce an id field in the books array. Each book has a unique id. In a real life application, this id could also be the database id. The point is that it should be unique for every item.

We also utilise this id in the each block as below.

{#each books as book, idx (book.id)}

Basically, the role of this id field is to help Svelte figure out which DOM node should be changed when the each block updates. After this modification, if we run the application and click the button, we will see proper data as below.

The book name and author name match as expected. The first book was successfully removed. However, Svelte only removed the appropriate DOM node and other nodes were left as they were before the update.

Conclusion

With this, we have learnt how to use Svelte Keyed Each Block. This is extremely useful when we have a requirement to update elements within our each blocks.

Hope this little post was useful. If you have any comments or queries about this post, please feel free to write them in the comments section below.

How to create and run your first Svelte Rollup Application?

Saurabh Dashora — Mon, 17 Jan 2022 06:35:02 +0000

In this post, we will learn how to run our first Svelte Rollup Application.

To make things easy to understand, we will create a small Svelte app from scratch and write a basic Rollup configuration to build and run the application.

But before that, let's go through some basics about Svelte.

1 – What is Svelte?

Svelte is a new front-end library. It is known for its blazing fast speed. Unlike other UI libraries such as React, Svelte is more of a compiler. While traditional frameworks like React or Vue do much of the work in the browser, Svelte shifts that work into the compile step.

Basically, browsers do not directly execute Svelte code. When we write Svelte code, we need to have a build step that translates that code into standard HTML, CSS & Javascript. This makes it faster while also transferring less code to the browser when the user makes a request.

2 – Setting up a new Svelte Project

To demo our Svelte Rollup application, we will create an extremely simple Svelte application. To achieve it, we have to first setup our project.

First, we will create a new directory to place our application files.

$ mkdir my-first-svelte-app

Next, we navigate into the directory and then initialize a new npm project.

$ cd my-first-svelte-app
$ npm init -y

The above commands will generate a package.json file for our application. Now we can install the required packages. As a first step, we install the svelte package.

$ npm install svelte

After the installation is completed, we are ready to write some code for our demo application.

3 – Creating the Svelte App

By convention, we put all our code into the src directory. You can use any editor of your choice or the terminal to create the same. Inside the src directory, we create two files App.svelte and main.js.

The App.svelte file will contain the actual component code as below:

`App.svelte`

<script>
    export let name;
</script>

<p>Hello {name}</p>

A very simple component indeed! We only have a paragraph that displays a hello message. The name within curly braces is a prop for the App component.

Next, we configure the main.js file as below:

`main.js`

import App from './App.svelte';

const app = new App({
    target: document.body,
    props: {
        name: 'Web Ninja'
    },
});

export default app;

Here, we create a new instance of the App component. We also specify that we want to load the component in the document body. Lastly, we supply the name as part of the props object.

With this our Svelte application is ready. Time to delve into how to actually run the application.

4 – Installing Svelte Rollup Packages

As discussed earlier, we will be using Rollup to build and run our application.

But what exactly is Rollup?

Rollup is basically a module bundler for Javascript applications. It takes our application code and bundles it into files that browsers can easily parse. In other words, Rollup converts our .svelte files into browser-understandable HTML, CSS and Javascript.

To get started with Rollup, we first need to install the required package.

$ npm install -D rollup

Please note that this is a development dependency.

Next, we also need to install two more Rollup development dependencies. See below:

$ npm install -D @rollup/plugin-node-resolve rollup-plugin-svelte

The @rollup/plugin-node-resolve package helps Rollup resolve any third party plugins. The second dependency rollup-plugin-svelte is a third party plugin that helps Rollup understand how to handle Svelte applications.

As with rollup, both of these dependencies are also development only.

5 – Rollup Configuration File

Rollup also requires a configuration file to understand what it needs to do with our source code.

We will start by creating a very simple Rollup configuration file. It will simply bundle our Svelte application file into the public/build folder.

See below:

`rollup.config.js`

import svelte from "rollup-plugin-svelte";
import resolve from '@rollup/plugin-node-resolve';

export default {
    input: 'src/main.js',
    output: {
        file: 'public/build/bundle.js',
        format: 'iife',
        name: 'app',
    },
    plugins: [
        svelte({
            include: 'src/**/*.svelte',
        }),
        resolve({ browser: true }),
    ],
}

Let us understand what we are doing here.

In line 1 & 2, we have the necessary imports.
In line 5, we specify the path to the input file. This is the main.js file of our application where instantiate our component.
In line 6, we specify the output. Basically, this is the destination of our built artifacts. We also specify the format as iife (immediately invoked function express). Along with this, we specify the name of the variable to which we assign the iife output value. In this case, we name the variable as app.
Lastly, from line 11 onwards, we specify the other configuration items. Basically, we tell Rollup the path to our .svelte files. Also, we indicate that we are building our application to work on a browser.

6 – Adding Rollup to NPM Script

The next step is to add Rollup to our NPM scripts so that we can build and run our application.

To run Rollup, we add a new script to our package.json file.

`package.json`

"scripts": {
    "dev": "rollup -c -w",
}

The -c flag specifies that we want Rollup to use a configuration file. By default, it checks for a file named rollup.config.js. We already created the file in the previous section.

The -w flag instructs Rollup to watch our files for changes. In case we make changes to our source code, Rollup will automatically rebuilds the artifacts.

We can now run npm run dev to build our Svelte application. The built files will be placed inside /public/build folder.

7 – Serving the Content

Though we have successfully compiled our application, we still need a way to serve it. Therefore, we create an index.html file in the public folder.

`index.html`

<html>
    <head>
        <meta charset="UTF-8" />
        <meta http-equiv="X-UA-Compatible" content="IE=edge" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <title>My First Svelte App</title>
        <script defer src="build/bundle.js"></script>
    </head>
    <body></body>
</html>

Important thing to note is the script tag where we specify the location to our newly built bundle.js file.

To serve the application, we will install a simple web server available with the sirv-cli package.

$ npm install -D sirv-cli

Finally, we add a script to run the application.

`package.json`

"scripts": {
    "dev": "rollup -c -w",
    "start": "sirv public"
},

We can now start our application by running the command npm run start. The application will be accessible on http://localhost:5000. You should see the hello message in the browser window.

Conclusion

With this, we have successfully learnt how to create and run a Svelte Rollup application. We created a new Svelte app from scratch and also wrote a basic Rollup configuration.

The code for this post is available on Github .

In the next post, we will look at how to configure Svelte Rollup Hot-Reload feature so that we can improve the development experience.

If you have any comments or queries, please feel free to mention them in the comments section below.

How to implement NestJS Passport Authentication using Local Strategy?

Saurabh Dashora — Sat, 08 Jan 2022 03:35:29 +0000

In this post, we will learn how to implement NestJS Passport Authentication using the passport local strategy. Authentication is a key aspect of any production-level application. While there are many ways to handle authentication, one of the more popular ways is to use Passport.

If you are new to NestJS in general, you can start by going through this post on NestJS Basics. However, if you are mainly interested in learning how to get started with authentication in NestJS, you can continue with this post.

1 – What is Passport?

Passport is a popular NodeJS library. It is used for authentication purpose and comes along with several strategies. These strategies basically implement different authentication mechanism. For our example, we will use the local strategy. This strategy makes use of username and password.

Passport executes a series of steps as below:

Authenticating a user by its credentials. These credentials can be username/password, JWT tokens or any other identity token.
Manage authentication state by issuing a JWT token.
Attach information about the user to the Request object.
To make things easy, the @nestjs/passport module wraps the entire passport usage pattern into familiar NestJS constructs. This makes it extremely easy to implement NestJS Passport Authentication.

2 – Install Packages

To enable the local strategy, we need to install the required packages as below:

$ npm install --save @nestjs/passport passport passport-local
$ npm install --save-dev @types/passport-local

Passport provides the passport-local strategy that implements the username/password authentication. Also, we install the @nestjs/passport and the passport packages. Finally, the @types/passport-local package helps us with Typescript types during development.

Note - For any strategy, we need to always install the @nestjs/passport and the passport packages. The third package depends on the strategy we are trying to implement. For the local strategy, we install passport-local. Similarly, for the JWT strategy, we use the passport-jwt strategy. Even the types package depends on the strategy.

With the installation done, let’s now begin building our application.

3 – Creating the Modules

As a first step, let’s divide our application into appropriate modules. We will creating an auth-module and users-module. Basically, the auth-module will contain the logic for authenticating the user. On the other hand, the users-module will contain the user information.

With this view, we will create both the modules and their associated services as below:

$ nest g module auth
$ nest g service auth
$ nest g module users
$ nest g service users

Next step is to build our passport strategy. Configuring a strategy has two typical steps as below:

First is a set of options that are specific to a particular strategy. For example, the JWT strategy needs a secret to sign the tokens.
Second is a verify callback. In other words, we tell passport how to verify whether a user is valid. Passport expects this callback to return the full user object if the validation is successful. Also, it should return null if validation fails. Failure can mean that the user does not exist. For a strategy like passport-local, it can also mean that the password is invalid.

The NestJS passport package helps with the above two steps by providing helper classes.

4 – Implementing the User Service

Let’s first create the user service. See below example:

import { Injectable } from '@nestjs/common';

export type User = any;

@Injectable()
export class UsersService {

    private readonly users = [
        {
            userId: 1,
            username: 'John Marston',
            password: 'rdr1',
        },
        {
            userId: 2,
            username: 'Arthur Morgan',
            password: 'rdr2',
        },
    ]

    async findOne(username: string): Promise<User | undefined> {
        return this.users.find(user => user.username === username)
    }
}

For our demo example, our user service will simply contain a list of hardcoded valid users. However, for a real application, we would probably create a user database and fetch the users from the appropriate table using NestJS TypeORM or NestJS Sequelize libraries. You can also using MongoDB using the NestJS Mongoose library.

In the above example, the findOne() method simply fetches the user from the users array. Also, we need to update the exports array for the UsersModule as below:

import { Module } from '@nestjs/common';
import { UsersService } from './users.service';

@Module({
  providers: [UsersService],
  exports: [UsersService]
})
export class UsersModule {}

5 – Implementing the Auth Service

As the next step, we implement the Auth Service in the Auth Module.

See below:

import { Injectable } from '@nestjs/common';
import { UsersService } from 'src/users/users.service';

@Injectable()
export class AuthService {
    constructor(private usersService: UsersService){}

    async validateUser(username: string, password: string): Promise<any> {
        const user = await this.usersService.findOne(username);

        if (user && user.password === password) {
            const {password, ...result} = user
            return result
        }
        return null
    }
}

Basically, the auth service has the job of retrieving the user and verifying the password. As you can see, we create the validateUser() method. We fetch the user by using the user service and return the user object as output. However, we strip the password property out of the object before returning it.

Finally, we update the AuthModule to import the UsersModule. Without this step, we will not be able to use the user service in the auth module.

import { Module } from '@nestjs/common';
import { UsersModule } from 'src/users/users.module';
import { AuthService } from './auth.service';

@Module({
  imports: [UsersModule],
  providers: [AuthService]
})
export class AuthModule {}

6 – Implement Passport Local Strategy

This is a key step in implementing the NestJS Passport Authentication.

Basically, we need to implement the passport local strategy. To do so, we create a file named local.strategy.ts in the auth folder.

import { Injectable, UnauthorizedException } from "@nestjs/common";
import { PassportStrategy } from "@nestjs/passport";
import { Strategy } from "passport-local";
import { AuthService } from "./auth.service";

@Injectable()
export class LocalStrategy extends PassportStrategy(Strategy) {
    constructor(private authService: AuthService) {
        super()
    }

    async validate(username: string, password: string): Promise<any> {
        const user = await this.authService.validateUser(username, password);

        if (!user) {
            throw new UnauthorizedException();
        }

        return user;
    }
}

Basically, we create a class LocalStrategy that extends the PassportStrategy class. We also pass the attribute Strategy in the class definition. Note here that the Strategy is imported from passport-local and not passport package.

In the constructor, we simply call the super() method. The local strategy only expects the username and password fields and therefore, configuration options are not needed. However, if needed, we can pass extra properties while calling super().

Next, we implement the validate() method as part of the PassportStrategy class. For every strategy, Passport will call the verify function. In NestJS, this function is implemented with the validate() method. Based on the strategy, it expects some arguments. For example, in the local strategy, it expects the username and password attributes.

From a logical point of view, this method is quite simple. It simply calls the validateUser() method from the Auth service. If a valid user is found, it returns the same. Otherwise, it throws an exception. Most of the heavy work is done in the Auth Service.

Lastly, we need to update the Auth Module as below to use the passport module.

import { Module } from '@nestjs/common';
import { PassportModule } from '@nestjs/passport';
import { UsersModule } from 'src/users/users.module';
import { AuthService } from './auth.service';
import { LocalStrategy } from './local.strategy';

@Module({
  imports: [UsersModule, PassportModule],
  providers: [AuthService, LocalStrategy]
})
export class AuthModule {}

7 – Creating the Login Route

Now, we can create the login route. Basically, this route will be used to authenticate a user.

import { Controller, Post, Request, UseGuards } from '@nestjs/common';
import { AuthGuard } from '@nestjs/passport';

@Controller()
export class AppController {

  @UseGuards(AuthGuard('local'))
  @Post('login')
  async login(@Request() req) {
    return req.user;
  }
}

As you can see, we use the standard @Controller() decorator. Also, we use @Post() for the login() request handler. If you wish to know more about creating controllers, you can refer to our detailed post on NestJS Controllers.

Important thing to note here is the @UseGuards(AuthGuard(‘local’)) decorator. Basically, the AuthGuard is a special guard provided by the @nestjs/passport package. This guard was provisioned when we extended the passport-local strategy.

This built-in guard invokes the Passport strategy and starts the entire process. In other words, based on the strategy in the AuthGuard (in this case, local), this guard will retrieve the credentials, run the verify function and create the user property.

The user can simply authenticate by using the /login route as below:

$ curl -X POST http://localhost:3000/auth/login -d '{"username": "John Marston", "password": "rdr1"}' -H "Content-Type: application/json"

In case the user is valid, we receive the user object in response. Else, we receive the HTTP Status 401 or Unauthorized.

Conclusion

With this, we have successfully learnt how to implement NestJS Passport Authentication using the passport local strategy.

We started by installing the required packages and then building the modules for retrieving users and authenticating them. Lastly, we implemented the login route to trigger the passport authentication process.

The code for this post is available on Github.

In the next post we will be implementing the NestJS Passport JWT Strategy.

If you have any comments or queries, please feel free to mention them in the comments section below.