DEV Community: Amanpreet Singh

MongoDB 8.0: An Overview of the Latest Features and Upgrades

Amanpreet Singh — Tue, 26 Nov 2024 13:07:50 +0000

Introduction

The recent release of MongoDB 8.0 continues its tradition of innovation, introducing enhancements that promise to revolutionize how developers interact with data. Whether you're a seasoned MongoDB user or exploring it for the first time, this latest version offers exciting opportunities to elevate your development.

MongoDB is used across various industries, from large-scale e-commerce platforms and stream processing to transactional systems and complex Internet of Things (IoT) networks.

Whats new in MongoDB 8.0?

Performance Updates

The most popular document database is now faster than ever,

With 36% quicker reads ,

59% higher throughput for updates ,

200% faster time-series aggregations ,

and up to 50 times faster resharding.

Logging

You can configure the DB Profiler to log slow operations based on the time that MongoDB takes to process an operation, rather than the total latency of the operation. This allows for more accurate logging of slow queries, as it focuses specifically on the time MongoDB spends processing them.
Slow query logs feature a metric called queues.execution.totalTimeQueuedMicros. This metric helps determine if an operation is slow due to the time it takes to complete or the time it spends waiting to start.
Query analysis tools like the Query Profiler, Performance Advisor, and Search Query Telemetry report slow operations based on workingMillis rather than durationMillis. This shift offers a more accurate view of problematic queries.

Aggregation

You can use $convert operator to convert string values to binData values and vice versa.
New helper expression, $toUUID which provides simplified conversions of strings to UUID values.
Behavior of $rank and $denseRank: In MongoDB 8.0, null and missing field values in $denseRank and $rank for sortBy operations are treated the same when calculating rankings, thus making the behavior of $denseRank and $rank consistent with $sort.

Security

Starting in MongoDB 8.0, Queryable Encryption supports the range queries on encrypted fields. You can use $lt, $gt, $lte, and $gte operators on encrypted fields.
Open Cybersecurity Schema Framework : In MongoDB 8.0, you can specify the OCSF schema for audit log messages.

{
     "activity_id" : <int>,
     "category_uid" : <int>,
     "class_uid" : <int>,
     "time" : <int>,
     "severity_id" : <int>,
     "type_uid" : <int>,
     "metadata" : <document>
     "actor" : {
        "user" : {
           "type_id" : <int>,
           "name" : <string>,
           "groups" : <array of documents>
        }
     }
  }

Ingress Queue : MongoDb 8.0 introduces a new queue for ingress admission control. Operations waiting to enter the database from the network are placed in the ingress queue.

By default, this queue is unrestricted, allowing all operations to proceed without any queuing. However, by setting a maximum limit for the queue, you can manage operations at this stage, queuing them if the number of concurrent operations reaches the specified limit.

Replication & Sharding

Majority Write Concern : In previous releases, write operations that use the majority write concerns would wait and return an acknowledgement after the majority of replicate set members applied the change.

Starting in MongoDB 8.0, these operations return an acknowledgement when the majority of replica set members have written the oplog entry for the change.
Moving a Collection : Starting in MongoDB 8.0, you can unshard collections and move unsharded collections between shards on sharded clusters.
Use of $lookup stage in Transactions : Starting with MongoDB 8.0, you can use the $lookup stage within a transaction when targeting a sharded collection.
Store Application Data on Config Shards : Starting in MongoDB 8.0, you can configure a config server to store application data in addition to the usual sharded cluster metadata. The config server is then known as a config shard.

General Changes

Bulk Write command: The existing bulkWrite() method only allows to modification of one collection in one request but from MongoDB 8.0, you can use the new bulkWrite command to perform many insert, update and delete operations on multiple collections in one request.
New Query Shape and Query Settings
Read Concern on Capped Collections
Parameter Filtering
Shutdown Performance
Sort Support for updateOne
Query Planner Optimization Time : The method explain() returns information on query planner optimization time. The queryPlanner.optimizationTimeMillis status shows the time in milliseconds that the query planner spent on optimizations.

Upgrading to MongoDB 8.0

To upgrade to MongoDB 8.0 from a 7.x deployment, the 7.x deployment must have featureCompatibilityVersion set to 7.x.

To check the version use the below command

db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

To download MongoDB 8.0, go to the MongoDB Download Center.

If you're interested in the latest features of MongoDB 7.0 instead of the newest version, you can find the information here.

Conclusion

MongoDB 8.0 has certainly raised the bar. With a full set of new features, security improvements, and other performance enhancements, MongoDB 8.0 is the perfect choice for organizations looking to take their development to the next level.

I hope you have learned something new, as I did. If so, kindly like and share the article, and follow me to read more exciting articles.

References

MongoDB Docs

Release Notes for MongoDB

Essential Guide to Mastering Git Tags

Amanpreet Singh — Tue, 13 Aug 2024 08:30:21 +0000

Introduction

In software development, version control is essential for managing changes, tracking progress, and ensuring the integrity of your codebase. Git, the most popular and widely used is a distributed version control system, that offers various tools to help developers maintain a clean and organized project history. One such tool is Git tags. In this article, we'll explore what Git tags are, why they are important, and how you can use them to enhance your development process.

What are Git Tags?

Git tags are references to specific points in a Git repository's history. Git tags are fixed and immutable, unlike branches, which move as you continue to commit. Tags are typically used to mark important milestones, such as releases, breaking changes (e.g., v1.0.0), making it easier to refer back to them later.

Difference between Git tags and Git branches

Git Branches are mutable references whereas Git Tags are immutable references. In other words, a git tag is created to point to a specific commit and after that does not change, whereas a new commit will automatically update the branch to that commit.

Types of Git tags

There are two types of tags in Git - lightweight and annotated

Lightweight Tags

Lightweight tags as the name suggests are simple references to a commit. They act like a branch that doesn’t change, effectively serving as a quick pointer to a specific commit.

Lightweight tags are also considered as temporary tags as they are not stored in the Git database.

Lightweight tags can be created by using the following command git tag <tagname> as shown below.

git tag v1.0.0

Annotated Tags

Annotated tags are more detailed. They store additional information, including the tagger’s name, email, date, and a tagging message. They are stored as full objects in the Git database. They can also be signed and verified with GPG (GNU Privacy Guard).

Since annotated tags are detailed, they include extra information hence they are created while adding a message such as git tag -a <tagname> -m "message".

git tag -a v1.0.0 -m "Release version 1.0.0"

To create an annotated tag, use the -a flag and include a message with the -m flag together with tagname.

List and View Tags

To list all tags in the repository use git tag command.

git tag

# The above command will output something like this.
v1.0
v1.1
v1.2
v1.3
v1.4
v1.5

To get detailed information on a particular tag use git show <tagname> command.

git show v1.0

# The above command will output something like this.
commit 0e9c504d7ec2ee4cef250dff136826065fd8d9a6 (tag: v1.4)
Author: Aman <aman@example.com>
Date:   Fri Jul 12 19:12:40 2024 -0400
    feat: Add refresh token storage and validation

Above is an example of a lightweight tag displaying the information.

Sharing Tags

By default, tags are not pushed to the remote repository when you push commits. To push tags you need to explicitly use the command git push <remote> <tagname>.

git push origin v1.4

In case there are a lot of tags and you want to push it one go, you can use a --tags flag, here is the command for that. Using a --tags flag pushes all types of tags.

git push origin --tags

To push only annotated tags to the remote use the flag --follow-tags.

git push origin --follow-tags

Deleting Tags

To delete a local tag simply use git tag -d <tagname> as shown in the below command.

git tag -d v1.4

To delete a remote tag use git push <remote> -d <tagname> command.

git push origin -d v1.4

Git Tags Cheat Sheet

Command	Description	Example
`git tag`	Lists all tags in the repository.	`git tag`
`git tag <tagname>`	Creates a lightweight tag at the current commit.	`git tag v1.0.0`
`git tag -a <tagname> -m "<message>"`	Creates an annotated tag with a message.	`git tag -a v1.0.0 -m "Initial release version 1.0.0"`
`git show <tagname>`	Displays details of an annotated tag, including the message.	`git show v1.0.0`
`git tag -d <tagname>`	Deletes a local tag.	`git tag -d v1.0.0`
`git push <remote> <tagname>`	Pushes a specific tag to the remote repository.	`git push origin v1.0.0`
`git push --tags`	Pushes all local tags to the remote repository.	`git push --tags`
`git push <remote> --follow-tags`	Pushes only annotated tags to the remote repository.	`git push origin --follow-tags`
`git push origin -d <tagname>`	Deletes a tag from the remote repository.	`git push origin -d v1.0.0`
`git tag <tagname> <commit>`	Tags a specific commit instead of the latest one.	`git tag v1.0.0 9fceb02`
`git checkout <tagname>`	Check out a tag (detached HEAD state).	`git checkout v1.0.0`
`git tag -s <tagname> -m "<message>"`	Creates a signed tag using GPG.	`git tag -s v1.0.0 -m "Signed release"`
`git verify-tag <tagname>`	Verifies a signed tag.	`git verify-tag v1.0.0`

Best Practices for Git Tags

Use Semantic Versioning: Adopt a versioning strategy like semantic versioning (e.g., v2.1.0) to keep your tags meaningful. You can learn about semantic versioning here.
Tag Releases: Always tag a commit when you release a new version of your software. This can be automated via CI/CD pipelines based on your branching strategy.
Keep Tags Descriptive: Use annotated tags for significant commits to provide context and information about the tag. Annotated tags bring more information as compared to lightweight tags.

Conclusion

Git tags are a powerful tool in your version control toolkit, offering a simple yet effective way to mark important points in your repository's history. Whether you're managing releases, tracking milestones, or preparing for deployment, Git tags help you maintain a clear and organized workflow.

That's it for today, and congratulations to everyone who has followed this article! You've successfully learned about Git tags. Go and implement them in your projects and make your life as a developer easy! 🎉

I hope you have learned something new, just as I did. If you enjoyed this article, please like and share it. Also, follow me to read] more exciting articles. You can check out my social links here.

References

Git Documentation: Tagging

Top 9 Essential Utility Types in TypeScript

Amanpreet Singh — Mon, 12 Aug 2024 17:29:42 +0000

Why use TypeScript?

Typescript is a strongly typed superset of Javascript that gives you better tooling at any scale, enabling developers to catch errors at compile time rather than runtime.

TypeScript brings

Type Safety
Scalability
Enhanced IDE support

What are Utility Types?

Utility types are predefined types that help manipulate and transform other types. They help in simplifying complex type definitions.

By understanding and effectively using these utility types, you can write more concise, readable, and maintainable code, making your development process smoother and more efficient. These utilities are available globally.

Let's explore the top 9 utility types in TypeScript, with examples and explanations of how they can streamline your code.

Partial<Type>

Constructs a type with all properties of Type set to optional.

interface User {
  name: string;
  age: number;
}

const updatedUser: Partial<User> = {
  name: "Aman"
}

In the above code, the interface User has two required properties. However, by using the Partial<User> type, we can make these properties optional.

This can be useful in update operations where all the fields are not provided.

Required<Type>

Constructs a type containing all properties of Type set to required. You can consider this the opposite of Partial<Type>

interface User {
  name?: string;
  age?: number;
}

const newuser: Required<User> = {
  name: "Aman"
}
// error as `age` is missing in the newuser

The above code will give you the following error -

Property 'age' is missing in type '{ name: string; }' but required in type 'Required<User>'.

This utility is beneficial where all the properties are required.

Readonly<Type>

This utility constructs a type with all the properties of the Type sets to readonly, meaning the properties of the newly constructed type cannot be reassigned.

In other words, all the properties of a type Type are immutable.

interface User {
  name: string;
  age: number;
}

const user: Readonly<User> = {
  name: "Aman",
  age: 30,
};

user.name = 'Naman'; 
// Error: Cannot assign to 'name' because it is a read-only property.

This utility type can be used to avoid and protect objects from accidental mutations.

Record<Keys, Type>

Constructs an object type whose property keys are Keys and whose property values are Type. This utility helps map the properties of a type to another type.

type Role = "admin" | "user" | "moderator";
type Permission = "read" | "write" | "delete";

const roles: Record<Role, Permission[]> = {
  admin: ["read", "write", "delete"],
  user: ["read"],
  moderator: ["read","write"],
};

This utility helps map the properties of a type to another type.

Pick<Type, Keys>

Constructs a type by picking the set of properties Keys from Type

interface User {
  name: string;
  age: number;
  email: string;
}

type UserContactInfo = Pick<User, "name" | "email">;

const contact: UserContactInfo = {
  name: "Aman",
  email: "aman@example.com",
};

The utility type Pick<T, K> is very useful when you only need a subset of properties from a larger type.

Omit<Type, Keys>

Constructs a type by picking all the properties of Type and then removing Keys. This type is an opposite version of Pick<T, K>.

interface User {
  name: string;
  age: number;
  email: string;
  password: string;
}

type PublicUser = Omit<User, "password">;

const publicUser: PublicUser = {
  name: "Aman",
  email: "aman@example.com",
  age: 38
};

The utility type Omit<T, K> is useful when you need to exclude sensitive or unnecessary properties.

NonNullable<Type>

Constructs a type by excluding null and undefined from the defined Type.

type Name = string | null | undefined;

const username: NonNullable<Name> = "Aman";

The NonNullable<T> type is useful for making sure a value is always present, which helps avoid potential runtime errors.

Exclude<Type, Union>

Constructs a type by excluding from Type those types that are assignable to Union.

type Status = "success" | "error" | "pending";
type ErrorStatus = Exclude<Status, "success" | "pending">;

const error: ErrorStatus = "error";

The constant error will have excluded value from the type ErrorStatus. This type is useful for refining types by excluding or removing specific values.

Extract<Type, Union>

Constructs a type by extracting from Type those types that are assignable to Union.

type Shape =
  | { kind: "circle"; radius: number }
  | { kind: "square"; x: number }
  | { kind: "triangle"; x: number; y: number };

type Design = Extract<Shape, { kind: "circle" }>

In the above code, the type Design will be the type { kind: "circle"; radius: number; }. The Extract<T, U> utility type can be used to isolate specific types or values from a union.

Final Thoughts

These utility types are powerful tools in TypeScript. They let you manipulate and transform types in a way that is both clear and concise.

The utility types are not limited to those mentioned above; there are a few more utility types you can use in your projects such as

ReturnType<Type>
Parameters<Type>
ConstructorParameters<Type>
and a few more

Conclusion

That's it for today, and congratulations to everyone who has followed this article! You've successfully learned about the top 9 utility types in Typescript. Go and implement them in your projects! 🎉

I hope you have learned something new, just as I did. If you enjoyed this article, please like and share it. Also, follow me to read more exciting articles. You can check out my social links here.

How to Easily Import Data from Word Documents into Your App: A Complete Guide

Amanpreet Singh — Mon, 15 Jul 2024 00:30:06 +0000

Learn how to import data from Word documents into your app using Python with this comprehensive guide

Introduction

Recently, I was involved in data migration for a client. The data mainly consists of exam questions and their explanations. The data was structured in (.xlsx) format but there was one problem with the content of the data.

Some of the questions included mathematical equations which was a problem for us as it could not be saved as a text format in the cell of the Excel document. Usually, the equations are added in shape format which was difficult to read programmatically.

Some of the equations were very complex e.g.

So, instead of saving the mathematical questions in Excel format, they used DOC format which was way easy as compared to adding equations in an Excel format.

There were around 1000 mathematical questions that included equations in it. One way was to copy/paste manually and the other way was to do it programmatically.

Being a software developer I preferred the second way and you will find that in the next few minutes how I was able to import the data from a Word document, but before that, we should understand the importance of importing and exporting data.

Why Import/export data?

Import and export of data play a significant role in today's world. Almost all businesses require a set of data to formulate growth strategies and enhance operations, analysis, and decision-making.

The key benefits of importing and exporting data are

Enhanced decision making
Data sharing and Collaboration
Data Visualization and Reporting
Accuracy and Reliability
and many more...

The most common formats to import data are CSV, XML, or JSON as they ensure compatibility across different systems and platforms.

However today we will discuss a different format i.e. Doc files or Word Documents which is mainly a word processing document format. DOC files are used to store data such as formatted text, images, tables, and charts. The most common DOC formats widely used are .doc or .docx and they are merely not just a text file but larger than that.

Setting up the environment

We will be using a Python library python-docx to read and write DOC files. This library works well with the .docx format. If you are having .doc format, you might first need to convert them to .docx formats either by using Microsoft Word or some conversion tool.

One good thing that was done was the Doc file was formatted properly since there were 1000 questions to be imported, and each question with its details was added to a separate table. This not only helped organize the document but also did help in importing the document.

Below is the sample image of the Word document which was used to be imported.

Install python-docx

python-docx is a library for reading, creating, and updating (.docx) files. Let's first install the library.

pip install python-docx

The whole task of extracting data from a Word document was to

Read the Document i.e. use the library python-docx to extract text from the document.
Parse the extracted text into structured JSON
Generate JSON object.

Extracting Data

Read the Document

Below is an example of how to open a document file and read its content.

import docx

# Load the document
doc = docx.Document('sample.docx')

Structure of the Document

The basic structure of the document was in a tabular format, and it looks something like below.

No	Question	Solution
1	cos -640° =?	cos 80°
2	= ?	½ log 2
3	If tan² x + (1 - ) tan x - = 0 then x is	n - /4 or n + /3
4	The distance between the foci of the ellipse 4x²+5y²=20 is	2
5	If the two circles x²+y²+2gx+2fy=0 and x² + y²+2g1x+2f1y=0	f/f1=g/g1

Now based on the above structure we can easily identify that the first row is the header row and the rest of the rows are the data rows. In the next part, we will parse the data based on the tabular format.

💡 Keep in mind that if your mathematical equations are saved as plain text, they will appear as a regular string. However, if they are stored as LaTeX content, they will display exactly as they were saved.

Parse the Document

Let's parse the data based on the structure. Since our data is stored in tabular format, it is good to make sure that the count of tables in the document is 1 and not more than that.

# Count the No of Tables in the document.
table_count = len(doc.tables)
print("Number of tables in the document:", table_count)

In case it is more than 1 then you need to make sure about the structure of the table and whether it is to be extracted or not.

Let's extract the questions from the table, assuming we have multiple tables and the table to be used is at the index 0.

# Extract questions from the document
questions = extract_questions_from_table(doc)
print(questions)

def extract_questions_from_table(doc):
    return [
        {
            "index": row.cells[0].text,
            "question": row.cells[1].text,
            "solution": row.cells[2].text,
        }
        for row in doc.tables[0].rows[1:]
    ]

In the above code, we are using nested list comprehension. We can extract the data from the table. The outer loop for the table in doc.tables iterates over each table in the document, and the inner loop for row in table.rows[1:] iterates over each row in the current table, starting from the second row.

For each row, a dictionary is created with the text from the first three cells and added to the list.

The output should be something like this

[{'index': '1', 'question': 'cos -640° = ?', 'solution': 'cos 80°'}, 
{'index': '2', 'question': '= ?', 'solution': '½ log 2'}, 
{'index': '3', 'question': 'If  tan2 x + (1 - ) tan x -  = 0 then x is', 'solution': 'n - /4  or n + /3'}, 
{'index': '4', 'question': 'The distance between the foci of the ellipse 4x2+5y2=20 is', 'solution': '2'}, 
{'index': '5', 'question': 'If the two circles x2+y2+2gx+2fy=0 and x2 + y2+2g1x+2f1y=0', 'solution': 'f/f1=g/g1'}]

Generate the JSON from extracted data

Next, we can create the JSON format from the output we received earlier, by using the below code. Make sure the native module json is imported initially.

# Generate JSON based on the output response.
import json

def generate_json(questions, filename):
    questionnaires = []
    for question in questions:
        questionnaire = {
            "index": question["index"],
            "title": question["question"],
            "explanation": question["solution"],
        }
        questionnaires.append(questionnaire)
    with open(filename, "w") as f:
        json.dump(questionnaires, f, indent=4)

# Generate JSON File
jsonfile = generate_json(questions, "sample.json")

And voilà! We have successfully extracted the data from a Word document into JSON format. It is important to note that the more structured the data is, the easier it will be to extract.

The above data can be easily stored in any type of persistent storage, such as RDBMS or NoSQL databases.

Conclusion

That's it for today, and congratulations to everyone who has followed this blog! You've successfully imported data from a structured Word document into JSON format. Awesome job! 🎉

I hope you have learned something new, just as I did. If you enjoyed this article, please like and share it. Also, follow me to read more exciting articles. You can check out my social links here.

Why Every Web Developer Should Know HTMX?

Amanpreet Singh — Fri, 22 Sep 2023 06:03:27 +0000

Introduction

In an era dominated by front-end frameworks like React, Angular, Vue, and Svelte simplicity can often feel like a lost art. Enters HTMX - a lightweight, yet powerful, javascript library that dares to reimagine the way we think about modern web development.

So what is HTMX?, lets deep dive into the world of HTMX for the next few minutes and explore everything about its key features and breathing new life into the increasingly complex landscapes of front-end development.

What is HTMX?

What if you could supercharge your HTML with AJAX requests, CSS transitions, web socket, and server-side events capabilities directly from HTML attributes? That's what HTMX can do for you.

HTMX is small (approx 14k min.gz in size), dependency-free, extendable, and IE11 compatible. Yes you heard me right it is compatible with IE11

Why Choose HTMX?

Simplicity and Ease of Learning: HTMX allows you to add dynamic and interactive content directly within HTML without any need to write custom JavaScript. Making it easier for developers who are more comfortable with HTML and CSS.

It's especially helpful for newcomers to web development or server-side developers wanting to add a little front-end magic.
Server-Side Rendering Friendly: HTMX is designed to work well with server-renders HTML, making it a good fit for projects that rely on server-side frameworks like Django, and PHP.
Rapid Prototyping: The simplicity of HTMX makes it well-suited for quickly prototyping dynamic behavior without having to set up a full front-end development environment.

Core Concepts and Terminology

HTMX makes HTML extensible, more interactive and thus turning static HTML into "hypertext", which is HTML with behaviors and dynamic capabilities.

AJAX: HTMX makes it easier to use AJAX right within HTML, without writing JavaScript code
WebSockets: Support of Websockets in HTML via HTMX enables real-time data exchange with the server.
Server-Sent Events: HTMX supports the protocol for receiving updates from the server over a single HTTP connection, making it easier to build real-time updates into your web app.
hx-* Attributes: The hx-* attributes are custom attributes introduced by HTMX to extend HTML tags with dynamic behaviors making them more interactive and extensible. Some of them are hx-get, hx-post, hx-target etc.
Event Handling: Event handling in HTMX allows you to manage the lifecycle of HTTP requests, from the moment the event is triggered to the time when the new content is swapped into the DOM, thus adding more interactivity and dynamic behavior to the applications.

Enough with the theoretical part, let's now deep dive into some practical work with some code examples. Before that let's set up HTMX to use further.

Setting up HTMX

HTMX is a dependency-free, browser-oriented javascript library, thus no need for complicated build steps or systems.

Via a CDN: The fastest and simplest way is to add the below code to the head tag of your web page.
Download a copy: The next simplest way is to download htmx.min.js from unpkg.com, add it to the appropriate directory in your project, and include it using <script> tag.
Via NPM: For NPM-based build systems, you can install HTMX via a simple command

Basic Examples and Use Cases

Button Click to Fetch Data

The below example shows how to use HTMX to fetch data from a URL when a button is clicked.

<button hx-get="/fetch/data" hx-trigger="click" >Fetch Data</button>

When the button is clicked, HTMX sends a GET request /fetch/data and replaces the content of the button with the response.

Updating a ToDo List

Suppose you have a ToDo list and you want to mark an item as completed when it's clicked.

<ul id="todo-list">
  <li hx-post="/mark-complete" hx-trigger="click" hx-swap="outerHTML">Buy Milk</li>
  <li hx-post="/mark-complete" hx-trigger="click" hx-swap="outerHTML">Walk the Dog</li>
</ul>

When you click an item, HTMX sends a POST request to /mark-complete and replaces the clicked li element with the new HTML received from the server.

Form Submission

The below example shows how to submit a form using HTMX

<form hx-post="/submit/form" hx-trigger="submit">
  <input type="text" name="username" placeholder="Enter your username">
  <button type="submit">Submit</button>
</form>

When the form is submitted, HTMX sends a POST request to /submit/form with the form data. The form content will be replaced by the server's response.

Some of the other Use Cases can be like:

Live Search
Infinite Scroll
Form Validation
Bulk update
Real-time updates

Advanced Features

HTMX offers several advanced features that make it a powerful tool for developing interactive and dynamic web applications. Below are some of the features.

Partial Updates with `hx-select`

the hx-select attribute allows you to specify which part of the server's response should be used to update the DOM. The enables you to perform partial page updates and reduce the amount of data transferred over the network.

<div hx-get="/fetch/data" hx-select="#some-element">
  Click to Fetch Partial Data
</div>

Request Indicators

HTMX allows you to add visual indicators that show when a request is in progress. The hx-indicator attribute lets you specify an element that will be displayed during the AJAX call.

<button hx-get="/data" hx-indicator="#loadingSpinner">
    Fetch Data
</button>
<div id="loadingSpinner" style="display:none;">Loading...</div>

Trigger Modifiers

HTMX supports various trigger modifiers like once, delay, and throttle to fine-tune when and how often should be triggered.

<button hx-get="/data" hx-trigger="click delay:500ms">Fetch Data</button>

Lazy Loading

With the hx-trigger attribute set to revealed, HTMX enables lazy loading of content as elements come into view, this can improve performance by reducing initial load times.

<div hx-get="/lazy-load" hx-trigger="revealed">
  <!-- Content will be loaded when this element comes into view -->
</div>

History Management

HTMX provides a way to manipulate browser history. With attributes like hx-push-url, you can change the URL displayed in the browser's address bar after a request, facilitating navigation and bookmarking.

<div hx-get="/new/page" hx-push-url="true">
  Go to New Page
</div>

Based on the above code, when a user clicks on the div, HTMX will snapshot the current DOM and store it before it requests the/new/page. It then does the swap and pushes a new location onto the history stack.

WebSockets

If you wish to establish a WebSocket connection in HTMX, you use the ht-ws attribute.

<div hx-ws="connect:wss:/chatroom">
    <div id="chat_room">
        ...
    </div>
    <form hx-ws="send:submit">
        <input name="chat_message">
    </form>
</div>

The connect declaration establishes the connection and the send declaration tells the form to submit values to the socket on submit.

Comparing HTMX with Other Front-end Libraries/ Frameworks

As compared to some heavyweight frontend frameworks like Angular, React, or Vue, HTMX offers

Simple Learning Curve : HTMX has a significantly lower learning curve compared to frameworks like React, Angular, or Vue.
Performance : Since HTMX minimizes the need for client-side JavaScript, web pages often load faster and consume less memory.
Server Rendering : HTMX naturally embraces server-rendered HTML, making it a good fit for those who prefer this approach.

Integration with Backend Technologies

HTMX is backend-agnostic and can integrate seamlessly with virtually any server-side language or framework such as Django, Ruby on Rails, Flask, and others.

You can easily couple RESTful APIs or GraphQL endpoints, giving you flexible options for data fetching and manipulation.

HTMX native support of WebSockets and Server-Side events allows integration of real-time features.

Community & Ecosystem

While growing in popularity, its community is smaller compared to established frameworks like React, Angular, Vue, or Svelte. With vast community and documentation resources, the above-mentioned frameworks are widely used and are a go-to choice for many developers.

As HTMX attracts more followers, enthusiasts, and contributors, its community will grow naturally. It is still very young as compared to other front-end frameworks.

🚀 HTMX has been accepted into the first class of the GitHub Open Source Accelerator for 2023

Best Practices and Tips

Server-side validation: Take advantage of HTMX's server-side-centric model to handle data validation on the server, making the client lighter and more secure.
Utilize Event Hooks: Make the most of HTMX's lifecycle events for fine-grained control over requests and DOM updates.

Limitations of HTMX

Like any other tool, library, or framework we use, HTMX also has limitations, and some of them are listed below.

Limited tooling and No Type safety can make debugging a lot more difficult as compared to other frameworks.
"JS-in-HTML" approach has been in development earlier also with JS libraries and frameworks like Alpine, Knockout
Not ideal for SPAs
Lack of Virtual DOM: HTMX does not use virtual DOM, which can sometimes make complex UI updates less efficient.
No architectural approach for complex applications.

Conclusion

While growing in popularity, HTMX is carving its niche, offering simpler yet powerful alternatives to other widely used frontend libraries and frameworks. It is quite good for smaller projects and in the end, everything comes down to the project requirements and its complexity.

I hope you have learned something new as I did. If so, kindly like and share the article and also follow me to read more exciting articles. You can check my social links here.

References

HTMX Docs & Examples

Unpacking the Latest Features and Enhancements of MongoDB 7.0

Amanpreet Singh — Sat, 19 Aug 2023 06:03:51 +0000

Introduction

The recent release of MongoDB 7.0 continues this tradition of innovation, introducing a series of enhancements that promise to revolutionize the way developers interact with their data.

Whether you are a seasoned MongoDB user or exploring it for the first time, the latest version offers exciting opportunities to take your development to the next level.

MongoDB has found its way into various industry applications, from large-scale e-commerce platforms, Stream Processing, Transactional, and Analytical to intricate Internet of Things (IoT) networks.

What's new in MongoDB 7.0?

New Aggregations Operators

MongoDB 7.0 brings two new aggregation operators $median and $percentile

$median : In a dataset, the median is the percentile value where 50% of the data falls at or below that value.
$percentile : In a dataset, a given percentile is a value where that percentage of the data falls at or below that value.

If you are looking to learn more about aggregations, you can check this article with sample datasets top 5 essential aggregations of MongoDB

Time Series Collections

Time series data is a sequence of data points, typically measures at successive points in time. for e.g. stock prices, temperature readings.

Time series collections in MongoDB are specifically designed to handle time series data. These are optimized for storing, querying, and processing timestamped data.

Time series collections were first introduced in version 5.0 of MongoDB and were having some limitations. The new MongoDB version 7.0 removes most of the time series limitations that were based on the $delete command.

Together with this, it also brings significant improvements to working with time series data which results in improved storage and query performance.

Compound Wildcard Indexes

MongoDB 7.0 supports creating wildcard indexes on a field or a set of fields. A compound wildcard index has one wildcard term and one or more additional index terms.

To create a wildcard index on a single field use the following syntax:

db.collection.createIndex( { "<field>.$**": <sortOrder> } )

To create a wildcard index on all fields excluding _id we can use the following syntax:

db.<collection>.createIndex( { "$**": <sortOrder> } )

Atlas Search Index Management

MongoDB's Atlas Search feature allows fine-grained text indexing and querying of data. This feature is only available on Atlas Clusters.

Atlas Search index is a data structure that categorizes data in an easily searchable format. In simple words, it is the mapping between terms and documents that contain these terms.

In MongoDB 7.0 we can now manage Atlas Search Indexes with mongosh methods and database commands.

Large Change Stream Events

Change Streams allow applications to access real-time data changes without complexity and risk of tailing the operation log i.e oplog.

Change Streams can be beneficial for reliable business systems architectures, informing downstream systems once data changes are durable.

In the new version of MongoDB, if you have change stream events larger than 16MB, we can use the new $changeStreamSplitLargeEvent stage to split events into smaller fragments.

Security

Starting in MongoDB 7.0, security capabilities have been strengthened by making general availability of Queryable Encryption.

Queryable Encryption offers end-to-end encryption of sensitive data while preserving the ability to run equality queries on that encrypted data.

Working Model of Queryable Encryption:

The latest version of MongoDB Enterprise version supports OIDC i.e. OpenID connect authentication.

Upgrading to MongoDB 7.0

To upgrade a standalone from a previous major release, kindly follow the guidelines mentioned here

If you are trying to install the latest version of MongoDB, visit this link

At the time of writing this article, MongoDB 7.0 was not available via Docker Hub but should be available in near time.

Conclusion

With a full set of new features, security improvements, and other enhancements, MongoDB 7.0 is the perfect choice for organizations to take their development to the next level.

I hope you have learned something new as I did. If so, kindly like and share the article and also follow me to read more exciting articles. You can check my social links here.

References

MongoDB Docs

Release Notes for MongoDB

Queryable Encryption

General Acknowledgment

The images used to describe the Atlas Search Index and Model of Queryable Encryption have been taken from the MongoDB website and are owned by MongoDB Inc.

Mastering MongoDB: Top 5 Essential Aggregation Pipelines

Amanpreet Singh — Thu, 10 Aug 2023 08:53:53 +0000

Introduction

MongoDB is a leading NoSQL database with a schema-less nature, and the ability to handle large volumes of data for modern web applications.

Today MongoDB is been used in:

Streamline building AI-enriched applications.
Video Games that are global, reliable, and scalable.
Real-time analytical views of all the most important data.
Analyze and act on data from the physical world i.e. Internet of Things (IoT).
Store, edit, and present all types of content management.

Prerequisites

Basic knowledge of MongoDB. You can install MongoDB version 6.0 from here and can also use the MongoDB Atlas to try out these examples.

You can run the examples by using the MongoDB Playground.

What is Aggregation and its pipelines?

Aggregation:

MongoDB inherits capabilities to perform complex transformations and calculations, extracting meaningful information we need to aggregate data.

Aggregation simply refers to the process of collecting and processing data to produce a single consolidated output or result. E.g. calculating the sum, average, count, or other statistical measures from the dataset.

Aggregation Pipelines:

In MongoDB, the aggregation pipeline processes documents i.e. the data records, and returns computed results. A pipeline is made up of one or more stages, and each stage transforms the data i.e. documents as it moves through the pipeline.

The basic flow for any aggregation looks something like this as shown in the image below.

This framework allows for flexible, complex, and powerful operations on data, enabling complex queries and data transformations to be performed directly in databases.

Top 5 Essential Aggregation Pipelines

The top 5 essential aggregation pipelines which every MongoDB developer should know are

$match
$group
$project
$lookup
$unwind

We will discuss each aggregation in detail - from prototype form i.e. syntax to examples in the following context. We will be using some sample datasets, so let's deep dive into this.

Sample Dataset

Let's imagine we are handling data for an e-commerce platform. Below is the sample dataset for the products and orders collection to understand the top 5 essential aggregation pipelines.

Sample Dataset for products collection:

// Products Collection
[
    {
        "_id": 101,
        "productName": "Laptop",
        "brand": "TechCorp",
        "specs": ["16GB RAM", "512GB SSD", "Intel i7"]
    },
    {
        "_id": 102,
        "productName": "Headphones",
        "brand": "AudioTech",
        "specs": ["Noise Cancelling", "Bluetooth", "8hr Battery"]
    },
    {
        "_id": 103,
        "productName": "Mouse",
        "brand": "TechCorp",
        "specs": ["Wireless", "Optical"]
    },
    {
        "_id": 104,
        "productName": "Keyboard",
        "brand": "TechWorks",
        "specs": ["Mechanical", "RGB Backlit"]
    }
]

Sample dataset for orders collection:

// Orders Collection
[
    {
        "_id": 1,
        "customerName": "Alice",
        "productId": 101,
        "quantity": 2,
        "price": 1000,
        "orderDate": "2023-08-01",
        "shippingCity": "New York"
    },
    {
        "_id": 2,
        "customerName": "Bob",
        "productId": 102,
        "quantity": 5,
        "price": 50,
        "orderDate": "2023-08-05",
        "shippingCity": "Los Angeles"
    },
    {
        "_id": 3,
        "customerName": "Charlie",
        "productId": 101,
        "quantity": 1,
        "price": 1000,
        "orderDate": "2023-08-05",
        "shippingCity": "New York"
    },
    {
        "_id": 4,
        "customerName": "Alice",
        "productId": 103,
        "quantity": 3,
        "price": 20,
        "orderDate": "2023-08-08",
        "shippingCity": "Boston"
    },
    {
        "_id": 5,
        "customerName": "Eve",
        "productId": 104,
        "quantity": 1,
        "price": 70,
        "orderDate": "2023-08-09",
        "shippingCity": "San Francisco"
    }
]

Please note: productId in orders collection refers to the product's _id

$match

Filters the documents to pass only the document that matches the specified condition(s) to the next pipeline stage.

The $match has the following prototype form:

{$match: { <query> } }

Now based on our sample data from orders collection, let's write a $match query to filter orders that are shipped from New York City.

Check the query below:

db.orders.aggregate([
{$match: { shippingCity: "New York" } }
)]

Based on the dataset of orders collection there will be two documents as a result. The result for the above aggregation pipeline will be as follows:

[
  {
    "_id": 1,
    "customerName": "Alice",
    "orderDate": "2023-08-01",
    "price": 1000,
    "productId": 101,
    "quantity": 2,
    "shippingCity": "New York"
  },
  {
    "_id": 3,
    "customerName": "Charlie",
    "orderDate": "2023-08-05",
    "price": 1000,
    "productId": 101,
    "quantity": 1,
    "shippingCity": "New York"
  }
]

$group

This stage separates documents into groups according to the "group key". The output is one document for each unique group key. A group key is often a field or group of fields e.g _id, productId

💡 $group does not order its output documents.

The $group stage has the following prototype form:

{
    $group: {
        _id: <expression>, // Group Key
        <field1> : { <accumulator1> : <expression1> }, // Optional
        ...
    }
}

Let's determine the total orders for each customer. Based on the orders collection this can be determined by the following query:

db.orders.aggregate([
{ 
    $group: {
        _id: "customerName",
        totalOrders: { $sum : 1 }
    }
}
])

The above query will give us the following result:

[
  {
    "_id": "Bob",
    "totalOrders": 1
  },
  {
    "_id": "Eve",
    "totalOrders": 1
  },
  {
    "_id": "Alice",
    "totalOrders": 2
  },
  {
    "_id": "Charlie",
    "totalOrders": 1
  }
]

You can see from the above results that the results are not in order and also only the _id i.e. customerName and the accumulators i.e. totalOrders are displayed as output.

$project

This aggregation is used to pass along the documents with the requested fields to the next stage in the pipeline. The specified fields can be existing fields from the input documents or newly computed fields.

The $project stage has the following prototype form:

{ $project: { <specification(s)> } }

Based on the orders collection, let's process the documents to get the total order value of each order.

In the orders collection, we have two fields price and quantity. Let's use the $multiply operator to get the total order value.

db.orders.aggregate([
{     
    $project : {
        orderValue: { $multiply: ['$quantity', 'price'] },
        customerName: 1
    } 
}
])

The output of the above query will be as follows: We can see from the results that only customerName, orderValue, together with _idfields are projected in the next pipeline stage.

[
  {
    "_id": 1,
    "customerName": "Alice",
    "orderValue": 2000
  },
  {
    "_id": 2,
    "customerName": "Bob",
    "orderValue": 250
  },
  {
    "_id": 3,
    "customerName": "Charlie",
    "orderValue": 1000
  },
  {
    "_id": 4,
    "customerName": "Alice",
    "orderValue": 60
  },
  {
    "_id": 5,
    "customerName": "Eve",
    "orderValue": 70
  }
]

💡 Suppression of the _id field can be done by using _id: 0 in the $project stage.

$lookup

One of the most important aggregations. If you have basic SQL knowledge, it performs a left outer join to a collection in the same database to filter in documents from the "joined" collection for processing.

The $lookup stage adds a new array field to each input document. The new array field contains the matching documents from the "joined" collection.

The $lookup has three types of prototype forms:

Equality Match with a Single Join collection : To perform an equality match between a field from the input documents with a field from the documents of the "joined" collection.
Join Conditions and Subqueries on a Joined Collection : To perform correlated and uncorrelated subqueries with two collections, and perform other join conditions besides a single equality match, use the below syntax.
Correlated Subqueries Using Concise Syntax : The below syntax is used which removes the requirement for an equality match on the foreign local fields inside of an $expr operator:

Let's write a query to join products and orders using $lookup. We are using the reference id productId from the orders collection

db.orders.aggregate([
    {
        $lookup: {
            from: "products",
            localField: "productId",
            foreignField: "_id",
            as: "productDetails"
        }
    }
]);

The above query will display the result as follows:

[
  {
    "_id": 1,
    "customerName": "Alice",
    "orderDate": "2023-08-01",
    "price": 1000,
    "productDetails": [
      {
        "_id": 101,
        "brand": "TechCorp",
        "productName": "Laptop",
        "specs": [
          "16GB RAM",
          "512GB SSD",
          "Intel i7"
        ]
      }
    ],
    "productId": 101,
    "quantity": 2,
    "shippingCity": "New York"
  },
  {
    "_id": 2,
    "customerName": "Bob",
    "orderDate": "2023-08-05",
    "price": 50,
    "productDetails": [
      {
        "_id": 102,
        "brand": "AudioTech",
        "productName": "Headphones",
        "specs": [
          "Noise Cancelling",
          "Bluetooth",
          "8hr Battery"
        ]
      }
    ],
    "productId": 102,
    "quantity": 5,
    "shippingCity": "Los Angeles"
  },
  {
    "_id": 3,
    "customerName": "Charlie",
    "orderDate": "2023-08-05",
    "price": 1000,
    "productDetails": [
      {
        "_id": 101,
        "brand": "TechCorp",
        "productName": "Laptop",
        "specs": [
          "16GB RAM",
          "512GB SSD",
          "Intel i7"
        ]
      }
    ],
    "productId": 101,
    "quantity": 1,
    "shippingCity": "New York"
  },
  {
    "_id": 4,
    "customerName": "Alice",
    "orderDate": "2023-08-08",
    "price": 20,
    "productDetails": [
      {
        "_id": 103,
        "brand": "TechCorp",
        "productName": "Mouse",
        "specs": [
          "Wireless",
          "Optical"
        ]
      }
    ],
    "productId": 103,
    "quantity": 3,
    "shippingCity": "Boston"
  },
  {
    "_id": 5,
    "customerName": "Eve",
    "orderDate": "2023-08-09",
    "price": 70,
    "productDetails": [
      {
        "_id": 104,
        "brand": "TechWorks",
        "productName": "Keyboard",
        "specs": [
          "Mechanical",
          "RGB Backlit"
        ]
      }
    ],
    "productId": 104,
    "quantity": 1,
    "shippingCity": "San Francisco"
  }
]

You can see from the above response that an additional field productDetails is added which itself is an array and includes the matching documents based on the "joined" collection.

$unwind

This aggregation deconstructs an array field from the input documents to output a document for each element.

Each output document is the input document with the value of the array field replaced by the element.

The prototype form used for $unwind aggregation is as follows:

{
  $unwind:
    {
      path: <field path>,
      includeArrayIndex: <string>, // Optional
      preserveNullAndEmptyArrays: <boolean> // Optional, default value: false
    }
}

Let's use $unwind aggregation on the Product specs field of products collection: The syntax for the query will be as follows:

db.products.aggregate([
    {
        $unwind: {path: "$specs" }
    }
]);

The response to using $unwind will be deconstructed form of specs array field:

[
  {
    "_id": 101,
    "brand": "TechCorp",
    "productName": "Laptop",
    "specs": "16GB RAM"
  },
  {
    "_id": 101,
    "brand": "TechCorp",
    "productName": "Laptop",
    "specs": "512GB SSD"
  },
  {
    "_id": 101,
    "brand": "TechCorp",
    "productName": "Laptop",
    "specs": "Intel i7"
  },
  {
    "_id": 102,
    "brand": "AudioTech",
    "productName": "Headphones",
    "specs": "Noise Cancelling"
  },
  {
    "_id": 102,
    "brand": "AudioTech",
    "productName": "Headphones",
    "specs": "Bluetooth"
  },
  {
    "_id": 102,
    "brand": "AudioTech",
    "productName": "Headphones",
    "specs": "8hr Battery"
  },
  {
    "_id": 103,
    "brand": "TechCorp",
    "productName": "Mouse",
    "specs": "Wireless"
  },
  {
    "_id": 103,
    "brand": "TechCorp",
    "productName": "Mouse",
    "specs": "Optical"
  },
  {
    "_id": 104,
    "brand": "TechWorks",
    "productName": "Keyboard",
    "specs": "Mechanical"
  },
  {
    "_id": 104,
    "brand": "TechWorks",
    "productName": "Keyboard",
    "specs": "RGB Backlit"
  }
]

Best Practices

Like any other high-performance tool, MongoDB also performs best when it is used properly. Here are some of the best practices for making your MongoDB perform best.

Schema Design : Use embedded documents for data that are used more often for read operations. Use of references for data that won't be accessed every time.
Indexes : Always index fields that are often required, for e.g productId in the orders collection.
Aggregation : Use $match early to reduce the amount of data that subsequent stages need to process.
Data Modelling : Avoid the usage of large arrays that will grow over time. For e.g Product Specs
Limit the Result : Use $limit the operator to limit the result from large datasets, wherever applicable.

More Aggregation Queries to Try Out

Do you want to try out more on aggregation, below are some of the queries for you to try out at your end.

Calculate Total Sales for Each Product Spec.
Group by date to find the total sales for each day.
Filter orders that exceed a certain price (let's assume $500).
Find out the total revenue from each city.
Sort customers based on the number of items they purchased.

Let me know the results in the comments section, you can also look into the solutions provided by me, check the References section, or click here.

How to run the examples

You can run the examples by using the MongoDB Playground. You can follow the instructions here to get started with MongoDB Playground.

Conclusion

MongoDB is a NoSQL document database for modern developers working on high-performance applications. With its JSON-like documents, MongoDB allows developers to model data however the application requires. With nesting arrays and embedded sub-documents, it allows the modeling of complex relationships between data.

I hope you have learned something new as I did. If so, kindly like and share the article and also follow me to read more exciting articles. You can check my social links here.

References

MongoDB Docs

Use MongoDB Playground to run the examples in your browser without installing anything.

You can find the sample datasets and also the solutions to examples in my repo https://github.com/amanpreet-dev/essential-aggregation-pipelines

Do not forget to give a star for the repo and also follow me on GitHub

Test-Driven Development (TDD) with TypeScript for Beginners

Amanpreet Singh — Wed, 09 Aug 2023 06:45:38 +0000

Introduction to TDD

Before we deep dive into Test-Driven Development or TDD, let me ask a simple question to you.

Why developers do not write Tests often?

I being a software developer for the past 15 years have been asked/questioned this by/to my colleagues, juniors, and manual QA team members. Here are some of the common reasons.

Time Constraints : Writing tests takes time and with developers under tight schedules usually skip tests to meet delivery timelines.
Misconceptions that Tests aren't necessary : Some developers believe that if they write code carefully, it won't have bugs which leads to overconfidence and perceptions that tests are unnecessary.
Lack of Awareness or Training : Not all developers are trained in testing or understanding the benefits of tests. Some might not be aware of tools, and frameworks used for specific languages or platforms.
Short-Term vs Long-Term Perspective : Developers should consider tests as an investment that pays off in the long run by reducing bugs and improving code readability.
Lack of Management Support : If management doesn't recognize or advocate for the importance of testing, developers aren't appreciated and thus it becomes a less likely routine among developers.

Test-Driven Development (TDD) is a software development methodology that emphasizes the importance of writing tests before writing the actual implementation code.

What is TDD?

Test-Driven Development or TDD is a software methodology that enables developers to design simple, clean, and tested code. It is a standard way of producing high-quality software which is incorporated with the best coding and design practices.

Here is a step-by-step overview of the TDD process:

Write a Failing Test : Write a test for the feature that you are about to develop and that doesn't work initially.
Write the Minimum Code to Pass the Test : Write the minimum amount of code needed to pass the test.
Run the Test : Execute the test. If it fails, modify the code and retest until it passes.
Refactor : Once the test passes, examine the code and see if it can be improved or optimized without changing its behavior. One of the most important steps is to ensure code quality and maintainability.
Repeat : Go back to the first step and repeat the process for the next piece of functionality or feature you want to develop.

Advantages of TDD in Software Development

Below are the precise points highlighting the advantages of Test-Driven Development (TDD)

Improved Code Quality : TDD enforces writing tests, leading to more reliable and bug-free code.
Faster Debugging : By catching defects early in the development process, TDD reduces debugging time and minimizes the impact of errors.
Confident Refactoring : Developers can confidently refactor code since existing tests ensure that the intended functionality remains intact.
Continuous Integration : TDD complements CI/CD pipelines by integrating automated tests, ensuring that the new changes don't break existing functionalities.
Better Collaboration and Increased Productivity : Well-defined tests facilitate communication between developers and testers to understand the expected outcomes. TDD leads to better code design, saving time and effort during later stages of development.

How TDD works?

"Red-Green-Refactor" is the mantra that summarizes the TDD cycle:

Red : Writing a failing test means seeing a red indication from your test runner.
Green : Here, we write the minimum amount of code to pass the test thus getting a green indication in our test runner.
Refactor : Clean up the code, and make it more readable, and efficient without making the test fail. After refactoring the test should still pass.

Importance of Test Coverage

Since code is written around tests, there is a higher chance of more code covered by tests. This leads to fewer bugs and easier maintenance.

Remember it is not about 100% code coverage where our tests don't make sense or are not working as per expectations.

Getting Started with TDD and TypeScript

Ok, a lot about TDD in theory, now let's get deep knowledge by actually doing some code-level examples. We will be using TypeScript and Jest as our testing frameworks.

I will be using NodeJs v20.2 for this project, although it should work with almost all stable node versions. All the code will be available in the Github repo shared under the References section.

Setting up a TypeScript Project

To get started, open your terminal and create a new folder named tdd-for-beginners (you can choose a name of your choice too) and move into that directory. Use the following commands:

mkdir tdd-for-beginners && cd tdd-for-beginners

Next, we will initialize our project using the following command:

npm init -y
# using -y flag tells npm init to automatically use the defaults.

by running the above command a new file will be created in the repository, named package.json

Install TypeScript and ts-node

Now, let's configure the TypeScript compiler. Run the following command inside your project directory to install TypeScript.

npm i -D typescript ts-node
# This will add the package under devDependencies

TypeScript is a typed superset of Javascript that compiles to plain Javascript. ts-node is a tool that allows you to run TypeScript code directly, without the compile step. Install them as development dependencies using the above command.

Create a tsconfig.json file:

TypeScript uses a file called tsconfig.json to configure the compiler options for a project. Let's create this file in the root of the project directory.

# Will create the tsconfig.json file with default options
npx tsc --init

Configuring Testing Environment

We will be using Jest as the testing framework. Let's install the required dependencies by using the following command:

npm i -D jest ts-jest @types/jest

Before we write our first test, we need to configure a few things. Let's do them below:

To specify jest as our test runner, update the package.json file as follows
Configure Jest: Create a jest.config.js file at the root of your project.
TypeScript Configuration: Add the include and exclude properties after compilerOptions

Example 1: Writing Our First Test for adding two numbers

We will use TDD methodology to add two numbers and write tests for them. Now let's write our first test before we deep dive into more complex functionalities.

Create a Typescript file for the function e.g addNumbers.ts:
Create a test file e.g addNumbers.spec.ts
Now run the tests by using the below command:
Now let's make the test pass by adding the required functionality in the addNumbers.ts file. This follows our Green pattern.

Now there is not much to refactor here, so we will try to cover this concept in other complex examples.

Example 2: Writing a Palindrome Checker

Let's create a simple utility function to check if the string is palindrome or not. We will first create two files palindrome.ts and palindrome.spec.ts

// palindrome.spec.ts 
import { isPalindrome } from './palindrome';

test('should return true for a palindrome', () => {
  expect(isPalindrome('racecar')).toBe(true);
});

test('should return false for a non-palindrome', () => {
  expect(isPalindrome('hello')).toBe(false);
});


// palindrome.ts
export function isPalindrome(str: string) {

}

Now when you run the command npm test you will see a failed test or something like this. Remember the RED pattern.

Defining the Test Case

Now let's fix our test and for that, we should understand what the requirements are or in this case, what is a palindrome string.

An inputted string reads the same forward and backward for e.g - "racecar"
The input string which is not the same when read in reverse, is not a palindrome. e.g "hello"
The function should ignore the case while determining whether the string is palindrome or not. e.g "Madam"
The function should handle spaces and recognize the input string accordingly. e.g "A man, a plan, a canal, Panama"
The function should consider special characters while determining palindrome strings. e.g "$#malyalam#$"
Some of the edge cases to be considered: A single character or an empty string is also considered a palindrome. for e.g "a", ""

So in total, there are six test cases for which we need to write a code and make the test pass.

Implementing the Function

Based on the above requirements, let's work on the code and check which test cases pass and which fail.

export function isPalindrome(str: string) {
  const reversed = str.split("").reverse().join("");
  return str === reversed;
}

Running the Test

Based on the test cases defined earlier, we have six test cases to check whether our code works as expected or not.

Now let's run the test with different input string values and see where our test fails.

test("should return true for a palindrome", () => {
  expect(isPalindrome("Madam")).toBe(true);
});

The above test fails as it doesn't consider case-sensitive, let's fix that in our code.

export function isPalindrome(str: string) {
  str = str.toLowerCase(); 
  const reversed = str.split("").reverse().join("");
  return str === reversed;
}

Now the changes made as per the above code make our first three test cases pass. You can check the image below.

Now Let's work on the other test cases. Suppose you change the input string to "A man, a plan, a canal, Panama", which is a palindrome string, lets's check our tests whether it gets passed or failed.

test("should return true for a palindrome", () => {
  expect(isPalindrome("A man, a plan, a canal, Panama")).toBe(true);
});

The above test result will fail and the reason for it is not considering spaces, commas i.e. non-word characters though the string actually is a palindrome string. We need to remove non-word characters from the string. Let's refactor the code to make it pass.

// Remove the non-word characters 
export function isPalindrome(str: string) {
  str = str.replace(/\W/g, "").toLowerCase();
  const reversed = str.split("").reverse().join("");
  return str === reversed;
}

Now by making the above changes our test is passed and it looks something like this. Remember the Green pattern

Now out of six defined test cases, Four have been passed successfully. Let's work on the remaining two test cases.

// Pass a single character or an empty string
test("should return true for a palindrome", () => {
  expect(isPalindrome("")).toBe(true);
});

The above code will work and make our test pass but what if we add some special characters in our input string e.g. "#$malayalam$#", this will also pass our test.

test("should return true for a palindrome", () => {
  expect(isPalindrome("$#malayalam#$")).toBe(true);
});

Thus all are test cases defined as earlier passes. The final test file to check all the test cases defined earlier will look something like this.

// All the test cases defined
test("should return true for a palindrome", () => {
  expect(isPalindrome("racecar")).toBe(true);
  expect(isPalindrome("Madam")).toBe(true);
  expect(isPalindrome("A man, a plan, a canal, Panama!")).toBe(true);
  expect(isPalindrome("$#malayalam#$")).toBe(true);
});

test("should return false for a non-palindrome", () => {
  expect(isPalindrome("hello world")).toBe(false);
  expect(isPalindrome("TypeScript")).toBe(false);
});

test("should handle empty strings", () => {
  expect(isPalindrome("")).toBe(true);
});

The above examples demonstrate how to use TDD to develop small functions with TypeScript. We can apply the same principles and practices to more complex projects, increasing our confidence in the code's correctness and maintainability.

💡 Remember, the key is to write tests first and then implement the functionality, incrementally building up your codebase.

It's worth noting that TDD is predominantly associated with unit testing, but the methodology can also be applied (with modifications) to higher levels of testing, such as integration or end-to-end tests.

Conclusion

Despite its benefits, TDD is a challenging way of developing high-quality software, especially for developers new to practice. It requires discipline to always write tests first and to resist the urge to jump directly into coding solutions.

Using the TDD approach encourages developers to think about their code's requirements, design, and functionality from the start, ultimately leading to better software design and fewer defects.

I hope you have learned something new as I did. If so, kindly like and share the article and also follow me to read more exciting articles. You can check my social links here.

References

Below is the Github repo with all the code and configuration files.

https://github.com/amanpreet-dev/tdd-for-beginners

I have committed the code based on various reference points, you can check the CHANGELOG and Commits reference accordingly.

Mastering the Art of Node Package Management

Amanpreet Singh — Tue, 08 Aug 2023 05:41:55 +0000

Introduction

Before we deep dive into the world of Node package manager, we need to understand and know what is a package manager.

In simple words, a package manager is a system or set of tools used to automate installing, upgrading, configuring, and using software or applications.

A package manager is not just limited to Node.js, in fact, the OS which we use on a daily basis also uses package managers. E.g.

For MacOS we use the brew package manager. Similarly for Ubuntu, we use the apt-get package manager, and for Windows, the package manager we use is Winget or Chocolatey.

A brief history of Node Package Managers

Node.js comes with a default package manager NPM i.e. Node Package Manager. NPM is the world's largest software registry and hosts more than a million packages that are used by javascript developers daily.

There are other package managers too such as Yarn introduced by Meta, in 2016. Then we have PNPM i.e. Performant Node Package Manager released in 2017.

Together with npm, yarn, and pnpm there were a few more node package managers introduced like tink and ied but later on, was discontinued.

All about the package.json file

The package.json file is a fundamental part of every Node.js project or application and is used by all package managers including NPM, Yarn, and PNPM. The file serves as the manifest of your project and contains metadata about the project and its dependencies.

Here is an example of what a package.json file might look like

{
  "name": "my-node-project",
  "version": "1.0.0",
  "description": "A sample Node.js project",
  "main": "index.js",
  "scripts": {
    "start": "node index.js",
    "test": "jest"
  },
  "dependencies": {
    "express": "^4.17.13"
  },
  "devDependencies": {
    "@types/express": "^4.17.13"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Here are some of the common properties you will see in a package.json file

name: The name of your project and the application.
version: The current version of the project.
description: A short description of the project.
main: The main entry point of your project. Usually, it is the first file that is executed for the application.
scripts: It is an object which holds all the script commands that are part of the application's lifecycle such as starting the application, building, testing, etc.
dependencies: These are the packages that are required for your application to execute. These are installed using npm install --save <package_name> E.g. express
devDependencies: Some packages are required for development purposes only. These are installed using npm i --save-dev <package_name> for e.g types@express
There are a few more fields in the package.json file like keywords, author and license etc.

NPM (Node Package Manager)

NPM manages all the packages and modules for Node.js and consists of the command line client npm. It gets installed into the system with the installation of Node.js. The required packages and modules in the Node.js project are installed, uninstalled, and removed using NPM.

Key Features

Package installation and management : Easily install, update, and remove packages with simple commands.
Dependency resolution : Automatically resolves and installs package dependencies.
npm scripts : Run predefined scripts to automate tasks, such as building, testing, and deploying your application.
Semantic versioning : Manage package versions using semantic versioning, making it easy to control and update dependencies.
Security and auditing : Scanning projects for vulnerabilities and automatically applying patches with npm audit.

Examples

To install, update or remove a package using NPM, simply run the following commands:

# For installing a package
npm install <package_name>

# For updating a package
npm update <package_name>

# To removing a package
npm uninstall <package_name>

# To install all dependencies
npm install

Yarn

If you are looking for fast, reliable, and secure dependency management, yarn is the ultimate choice.

Installation

To install Yarn, we can use npm which comes as the default Node.js package manager. Though they are a few alternatives too for installing Yarn. Let's find out them below.

Using npm: Installing yarn package globally allows us to use it across different projects via the command line.
For macOS: we can install it via Brew or curl. Let us check the commands to be used.
For Windows: You can install it by executing a .msi file, Link available here, or by using a package manager like Chocolatey.

All about the `yarn.lock` file

The file yarn.lock is created or updated whenever dependencies are added. By using a lock file and a deterministic installation algorithm, Yarn can ensure that the structure of the node_modules directory is the same in every environment, whether it's your local development machine or a production server.

Key Features

Fast & Efficient : Parallel downloads with offline caching and selective version inclusion makes managing packages efficient
Deterministic installs: yarn lock files ensures consistent dependency installations across different environments
Workspaces : Easily manages multiple packages in a single repository.
Improved Security : Enforces strict SSL & checksums which ensures the integrity of downloaded packages.
Compatibility : Yarn is compatible with the NPM registry, thus allowing access to millions of NPM packages.

Examples

To install a package using Yarn, run the following command:

# To add a new package 
yarn add <package_name>

# To update an already installed package
yarn upgrade <package_name>

# To remove a package
yarn remove <package_name>

# To install all dependencies for a project
yarn install

PNPM (Performant Node Package Manager)

In simple words, PNPM is a fast, disk-space-efficient package manager.

Installation

Using npm: To install PNPM, we can use npm which comes as the default Node.js package manager.
@pnpm/exe is packaged with Node.js into an executable, so it may be used on a system with no Node.js installed.
For macOS: we can install it via Brew. Let us check the commands to be used. Using the brew package manager:
For Windows: You can install using a package manager like Chocolatey or Winget.

All about the `pnpm-lock` file

Like other package managers, PNPM also creates a lock file pnpm-lock.yaml when you install packages. This file ensures that every install results in the exact same dependency tree, across all environments.

Key Features

Efficient package storage : Uses a global package store, saving disk space and reducing installation times.
Strict Dependency management : Prevents the accidental use of packages that are not explicitly listed in your project's dependencies.
Workspaces : Supports mono-repos and simplifies dependency management for multi-package projects.
Content addressable caches : Ensures packages are immutable and cacheable, improving performance and security.
Compatibility : Compatible with the NPM registry, and use most npm-compatible tools.

Examples

To install, update or delete a package using PNPM, run the following command:

# To add a new package 
pnpm add <package_name>

# To update an already installed package
pnpm update <package_name>

# To remove a package
pnpm remove <package_name>

# To install all dependencies for a project
pnpm install

Choosing the right package manager

There's no one-size-fits-all solution when it comes to package managers. The choice depends on your project's requirements and your personal preferences. Here are some guidelines to help you decide:

If you prioritize compatibility and the support of a vast ecosystem, NPM is a reliable choice.
If speed, security, and offline capabilities are important to you, consider Yarn.
If disk space and installation efficiency are your main concerns, PNPM might be the best option.

Feature Comparison

Feature	NPM	Yarn	PNPM
Workspace Support	✔️	✔️	✔️
Hoisted `node_modules`	✔️ (by default)	✔️	✔️
Isolated `node_modules`	✔️	✔️	✔️ (by default)
Managing node.js versions	❌	❌	✔️
Autoinstalling peers	✔️	❌	✔️
Lock file	package-lock.json	yarn.lock	pnpm-lock.yaml
Patching dependencies	❌	✔️	✔️
Dynamic package execution	✔️ via `npx`	✔️ via `yarn dlx`	✔️ via `pnpm dlx`
Content addressable storage	❌	❌	✔️
Zero-Installs	❌	✔️	❌
CLI available	✔️	✔️	✔️

Conclusion

Javascript package managers have expanded to offer developers a wide range of choices beyond NPM. While NPM remains the default for many, Yarn and PNPM are also widely used, each bringing its own unique advantages.

Choosing between NPM, Yarn, and PNPM should be based on the specific needs of your project or team.

I hope you have learned something new as I did. If so, kindly like and share the article and also follow me to read more exciting articles.

References

Clickjacking: The Web Attack You Need to Know About

Amanpreet Singh — Mon, 24 Jul 2023 11:22:12 +0000

Introduction

15 years in development, 0 focus on cybersecurity. Until Zoom rejected my app for clickjacking. Scary stuff.

💡 Stop falling for clickjacking scams

A few weeks ago, my Zoom app was rejected as it failed their Security and Privacy Compliance review.

I was a little bit worried and a bit disappointed with myself because having such a vast experience I never focussed on security, but this was the time to learn something new and also to make my web apps secure from Clickjacking attacks.

Below is the screenshot of the email from the Zoom team.

Luckily the Zoom team was very helpful and they shared a few resources with me to resolve the Clickjacking issue.

So what is Clickjacking? How can we prevent it? All these questions must be in your mind. So let's continue getting an in-depth analysis of Clickjacking.

What is Clickjacking?

Clickjacking is a web attack where malicious websites trick you into clicking on hidden elements. These actions could lead to malware downloads, sharing sensitive info, or making unwanted purchases.

Protect yourself by staying vigilant and avoiding suspicious links.

A working example of Clickjacking

Suppose we have a website with a login form that looks this this

<form action="/login" method="post">
    <label for="username">Username:</label>
    <input type="text" name="username" id="username"><br>
    <label for="password">Password:</label>
    <input type="password" name="password" id="password"><br>
    <input type="submit" value="Login">
</form>

An attacker can create a malicious webpage that loads your login form (mentioned above) in an iframe and overlays it with another element, such as a fake button that says Click Me to win a prize!

<!-- malicious webpage -->
<html>
  <head>
    <title>Clickjacking Demo</title>
    <style>
      iframe {
        position: absolute;
        top: 100px;
        left: 100px;
        width: 400px;
        height: 200px;
        opacity: 0.5;
      }
      #overlay {
        position: absolute;
        top: 150px;
        left: 150px;
        width: 200px;
        height: 100px;
        background-color: red;
        opacity: 0.5;
      }
    </style>
  </head>
  <body>
    <iframe src="https://yourwebsite.com/login"></iframe>
    <div id="overlay"></div>
    <script>
      // this code listens for clicks on the overlay element and forwards them to the hidden iframe
      document.getElementById("overlay").addEventListener("click", function() {
        document.getElementsByTagName("iframe")[0].contentWindow.document.forms[0].submit();
      });
    </script>
  </body>
</html>

Now, when a user visits the attacker's page and clicks on the overlay element, they are actually clicking on the hidden login form of your website, which the attacker can use to steal the user's credentials.

Clickjacking is also called UI redress attack because the attack tampers with the user interface of the exploited website

Unmasking Clickjacking: The Hidden Threat

Clickjacking attacks can be carried out using a variety of techniques, including iframes, invisible buttons, and social engineering.

Clickjacking attacks can be used to steal sensitive information, spread malware, or manipulate user behavior.

Now let's add preventive actions in our above code using X-Frame-Options header with the DENY or SAMEORIGIN option to prevent your login form from loading in an iframe on another domain or origin

To prevent clickjacking attacks, web developers can use various techniques such as

Implementing the X-Frame-Options HTTP response header:
For applications using nginx as a web-server
There is also 1 more option available ALLOW-FROM uri - This option is used when there is a specific URI that is allowed to frame your website.
Using Content Security Policy i.e. CSP

Check the implementation

You can check the implementation of clickjacking prevention with the following guidelines -

After adding the X-Frame-Options header in your server configuration and also setting the Content Security Policy (CSP) headers in your server
Access your website on a web browser
Open the browser's developer tools to access the network tab.
Browse to the page or action for which you want to check for clickjacking protection.
Look for corresponding HTTP response headers

By examining the HTTP response headers in the browser's tools, you can verify if clickjacking preventive measures are correctly applied.

😱 Unbelievable Clickjacking Cases!

Some of the famous Clickjacking attacks have been

Adobe fixes Clikcjacking flaw

Google patches Clickjacking bug

Facebook likejacking attacks continue with flesh appeal

Twitter Worm

Jotform

All the above examples of our some very very big companies indeed.

🔐 Protect Your Online Life: Enhance Web Security

In a hyper-connected digital era, where vulnerabilities lurk at every click, it's crucial to safeguard our online presence. Clickjacking attacks are not new and as a developer, we should take full responsibility in applying the prevention methods.

Well, luckily using the above-mentioned methods, I was able to verify my Zoom app and prevent it from being suspended.

Below is the email reference for it

Conclusion

Clickjacking is a persistent threat that demands continuous attention from web developers and users alike. By staying informed about the potential risks and adopting best practices for web security we can prevent Clickjacking attacks.

I hope you have learned something new as I did. If so, kindly like it or share it with others so they can also see it.

Unpacking the Latest Features of NestJS v10

Amanpreet Singh — Mon, 24 Jul 2023 07:30:09 +0000

Introduction

NestJS v10 is here! This progressive Node.js framework offers efficient, reliable, and scalable server-side app building.

As per the StackOverflow Developer survey, NestJS is one of the top 10 most popular backend frameworks in the world. Check the survey results here.

It has more than 58k Github stars on its repository.

What's New?

Recently NestJS has officially released v10 for building efficient and enterprise-grade, server-side applications. Let's deep dive and see what are the new features of the new version.

SWC Integration

SWC stands for Speedy Web Compiler. It is an extensible Rust-based platform that can be used for both compilation and bundling. To speed up your development process, using SWC with NestJs CLI is a great way.

Installation of required packages is as follows:

$ npm install --save-dev @swc/cli @swc/core

You can now use SWC with NestJs CLI as follows:

$ nest start -b swc
# or
$ nest start --builder swc

Another way to use SWC by default is via nest-cli.json file, as mentioned below:

{
    "compilerOptions": {
        "builder": "swc"
    }
}

By default, SWC does not perform any type-checking itself, so to enable this option we need to use --type-check flag, as mentioned below:

$ nest start -b swc --type-check
# or
$ nest start --builder swc --type-check

💡 SWC is approximately 20 times faster than the default TypeScript compiler.

Redis Wildcard Subscriptions

Added support for Redis wildcard subscriptions. This feature allows one to subscribe to all messages that match a given pattern.

To start using Redis-based microservices, install the required packages as follows:

$ npm install ioredis

By default wildcard subscription is false, to enable the wildcard subscription

const app = await NestFactory.createMicroservice(AppModule, {
  transport: Transport.REDIS,
  options: {
    host: 'localhost',
    port: 6379,
    wildcard: true, // this is the new option available
  },
});

With wildcards enabled, we can use glob-type patterns in our subscriptions, such as:

ne?tjs subscribes to nestjs, nextjs and nuxtjs
ne*tjs subscribes to netjs and neetjs

What's Improved or Changed?

Together with new changes, there are some improvements in the latest v10 of NestJS.

Changes to Cache Module

Caching is a great and simple technique to improve your app's performance.

The CacheModule is now available as a standalone package @nestjs/cache-manager. Earlier this package was available under @nestjs/common

To install the required packages:

$ npm install @nestjs/cache-manager cache-manager

Overriding Modules in tests

Overriding of modules in tests has been introduced in v10. This feature is useful when you want to mock the entire module all at once instead of mocking each provider individually.

CLI Plugins and TypeScript >= 4.8

NestJS CLI plugins will now require TypeScript >= v4.8. This is due to breaking changes in TypeScript v4.8.

Dropping Support for Node.js v12

Since Node.js v12 went end of life (EOL), to use the new features of NestJS v10, a minimum version of Node.js v16 or higher is required.

This also allows the NestJS package to be compiled as ES2021 by default which results in a smaller library size and better performance.

Migration Guidelines

To migrate your existing NestJS project to the latest version, follow the links available below:

Migration from v9 to v10

Migration from v8 to v9

Migration from v7 to v8

Conclusion

NestJS v10 brings a lot of new exciting and improved features as compared to its older versions. If you are using NestJS in your projects please share it via comments.

I hope you have learned something new as I did. If so, kindly like it or share it with others so they can also see it.

References

Below are some of the references which you can follow for more in-depth knowledge

Redis Microservices

SWC or Speedy Web Compiler

Recursively Moving Files in a Directory with Node.js and TypeScript

Amanpreet Singh — Fri, 23 Jun 2023 10:41:17 +0000

Introduction

Recently, my Marketing team was facing an issue with uploading the files on the server. The folder structure of their files was somewhat like the one below.

Folder A/
 Folder B
    Folder 1
       file 1
       file 2
       file n
    Folder 2
       file 1
       file 2
       file n

They had to select each file one at a time for some processing which was time-consuming as they had to select a folder within a folder, Multiple file selection was still possible but not with folders, they consulted with me for some suggestions and instantly I thought that making the files move to one folder will solve their issue.

Now moving the files from folder to folder recursively was not an easy task as there were around 1k files so I thought making this work programmatically will solve our issue and below is a summary of how I did this.

I used Node.js and Typescript for this as they have all the components which we need and are also very easy to understand.

Prerequisites

To get started, you'll need to have Node.js and npm (Node Package Manager) installed on your machine. If you haven't installed them yet, you can download and install them from the official Node.js website (https://nodejs.org/).

For this article, I am using Node v20.2.0

Step by Step guide

Once you have Node.js and npm installed, follow the step-by-step guide.

Create and Initialize a new Node.js project

Open a terminal, create a new directory where you want to store your project files, and run the following commands:

# Folder name can be anything but do make it easily readable.
mkdir recursive-files && cd recursive-files

# This command will create a new default package.json file in your project directory.
npm init -y

Install TypeScript and ts-node

TypeScript is a typed superset of Javascript that compiles to plain Javascript. ts-node is a tool that allows you to run TypSctipt code directly, without the compile step. Install them as development dependencies using the following command :

 npm i typescript ts-node -D

Create a tsconfig.json file:

The tsconfig.json file is used to specify the root files and the compiler options to compile the project.

For creating a tsconfig.json file use the following command:

# Will create the tsconfig.sjon file wit default options which can be modified as needed
npx tsc --init

Time to write the code:

Create a new .ts file in your project directory. You can use the following command to create a new file.

touch index.ts

Importing necessary modules:

We will be using fs and path modules from Node.js. The fs modules allow us to interact with the file system, whereas the path module allows us to work with file and directory paths.

import fs from 'fs'
import path from 'path'

Defining the `moveFiles` function:

The below function takes two arguments: sourceDir (the directory from which we want to move the files) and targetDir (the directory where you want to move the files). The function returns a Promise that resolves when all the files have been moved or rejected if there's an error.

function moveFiles(sourceDir:string, targetDir:string): Promise<void> {
    return new Promise((resolve), (reject) => {
        // Code for moving the files
    });
}

Reading the source directory & Iterating over the files:

We will be using fs.readdir function to read the contents of the source directory. This function takes a callback that is called with two arguments: an error (if there is one) and an array of filenames.

// Reading the files from source directory
   fs.readdir(sourceDir, (err, files) => {
      if (err) {
          reject(err)
          return;
      }
    console.log('List of files', files);
);

We're using Array.prototype.forEach to iterate over the array of filenames. For each filename, we're doing the following:

Constructing the old and new paths: We're using path.join to construct the full paths to the file in the source and target directories.
Checking if the file is a directory or a regular file: We're using fs.lstatSync to get the stats of the file, and then checking if it's a directory or a regular file with Stats.prototype.isDirectory and Stats.prototype.isFile.
If the file is a directory, we're calling moveFiles recursively to move the files to the subdirectory.
If the file is a regular file, we're using fs.rename it to move the file to the target directory. This function takes a callback that is called with an error (if there is one).

// Continue from the above code
     console.log('List of Files', files);

      files.forEach((file) => {
        const oldPath = path.join(sourceDir, file);
        const newPath = path.join(targetDir, file);
        const stat = fs.lstatSync(oldPath);

        if (stat.isDirectory()) {
          moveFiles(oldPath, targetDir)
            .then(() => {
              if (files.indexOf(file) === files.length - 1) {
                resolve();
              }
            })
            .catch((err) => reject(err));
        } else if (stat.isFile()) {
          fs.rename(oldPath, newPath, (err) => {
            if (err) {
              reject(err);
              return;
            }

            if (files.indexOf(file) === files.length - 1) {
              resolve();
            }
          });
        }
      });

Handling errors:

If there is an error at any point like reading the directory, getting the stats of a file, or moving a file, Promise will be rejected with the error.

Please note we are using fs.lstatSync it as a synchronous method. For the asynchronous version, we can use fs.lstat to avoid blocking the event loop.

Calling the `moveFiles` function:

moveFiles('SourceDir', 'TargetDir')
.then(() => console.log('All files moved successfully'))
.catch((err) => console.log(err))

Conclusion

The above script will recursively read all directories and files under the specified directory and will successfully move each file into one target folder as specified.

Please note the above code will not work for the files with the same name in different directories, the files will be overwritten and we need to add some logic to check if a file with the same name already exists in the target directory.

I hope you have learned something new as I did. If so, kindly like it or share it with others so they can also see it.

References

You can find the gist of the sample code here

DEV Community: Amanpreet Singh

MongoDB 8.0: An Overview of the Latest Features and Upgrades

Introduction

Whats new in MongoDB 8.0?

Performance Updates

Logging

Aggregation

Security

Replication & Sharding

General Changes

Upgrading to MongoDB 8.0

Conclusion

References

Essential Guide to Mastering Git Tags

Introduction

What are Git Tags?

Difference between Git tags and Git branches

Types of Git tags

Lightweight Tags

Annotated Tags

List and View Tags

Sharing Tags

Deleting Tags

Git Tags Cheat Sheet

Best Practices for Git Tags

Conclusion

References

Top 9 Essential Utility Types in TypeScript

Why use TypeScript?

What are Utility Types?

Partial<Type>

Required<Type>

Readonly<Type>

Record<Keys, Type>

Pick<Type, Keys>

Omit<Type, Keys>

NonNullable<Type>

Exclude<Type, Union>

Extract<Type, Union>

Final Thoughts

Conclusion

How to Easily Import Data from Word Documents into Your App: A Complete Guide

Introduction

Why Import/export data?

Setting up the environment

Install python-docx

Extracting Data

Read the Document

Structure of the Document

Parse the Document

Generate the JSON from extracted data

Conclusion

Why Every Web Developer Should Know HTMX?

Introduction

What is HTMX?

Why Choose HTMX?

Core Concepts and Terminology

Setting up HTMX

Basic Examples and Use Cases

Button Click to Fetch Data

Updating a ToDo List

Form Submission

Advanced Features

Partial Updates with hx-select

Request Indicators

Trigger Modifiers

Lazy Loading

History Management

WebSockets

Comparing HTMX with Other Front-end Libraries/ Frameworks

Integration with Backend Technologies

Community & Ecosystem

Best Practices and Tips

Limitations of HTMX

Conclusion

References

Unpacking the Latest Features and Enhancements of MongoDB 7.0

Introduction

What's new in MongoDB 7.0?

New Aggregations Operators

Partial Updates with `hx-select`

All about the `yarn.lock` file

All about the `pnpm-lock` file