DEV Community: Ali Amjad

A journey of creating a package on NPM ( Fast File Converter )

Ali Amjad — Mon, 29 Aug 2022 19:11:28 +0000

introduction

in this article, I would love to discuss the journey I had when building a package and publishing it npm ( node package manager ).

Yesterday, I published the Fast File Converter Package and it's an achievement that I can give some of what I have learned to the community as I've got many from there ( all of us do ).

We will be more focused on the codebase in this article along with tips and notes that may help you when you create your next package.

Pdf Excel Generator VS Fast File Converter

Last year, I have published a package under the name of "Pdf Excel Generator" and you can find the archived GitHub repository. As of today, the package has been deprecated on NPM.

Fast File Converter is an upgrade of Pdf Excel Generator, there were limitations with the PEG :

it was written in plain JS.
there was a lot of bad code in it and bad architecture.
it was supporting 2 types only ( PDF & Excel )
there was no room to grow.
bad input validations
inefficient documentation.

However, with the FFC, I have opened a door with more possibilities:

It's written with TS
easier to achieve great source code with TS with good practices and types
now it's supporting 7 diffrent types.
the code & the architecture that is written has a lot of room to grow and scale up in the future.
with TS deceleration files, you can enjoy input validations from TS.

Overall, that was a huge boost and improvement from the previous package to this one.

Fast File Converter(FFC) codebase architecture

The Above image is the folder architecture of FFC, it's open source under an MIT license and you can find it on GitHub.

Demo Folder :

if you want to create a package, you must provide some examples for developers to understand to use your package.

I've created a Demo Folder and I have added some examples in it that are suitable for the usage of my package, by "suitable" I mean that the examples can change according to your package, in my case that I have built an express application that runs a server to use this package and download it when you hit the example's routes.

Dist Folder :

As some of you may don't know what a dist folder is, the dist or destination folder is a directory that I have specified for the Typescript builds to be there, so basically it's the build of the Project folder.

however it may seem weird that I have committed this folder, usually, it's not recommended to push your build into GitHub but :

if your package is running on the browser ( client side ), you must have it there so that only JS runs on the browser
my Demo folder is not importing the package from the NPM but rather from the dist folder, so it can be used as a development tool as well to see your changes right away with the demos.

Lib Folder :

Lib or Library folder is the directory where all of our core TS files of the package's functionality lie in there.

and this can be the most important folder because usually you only code in this folder to change the source code of the package and make changes to it.

and at the end, TS files in this directory will be compiled to JS to be ready for production ready.

Script Folder :

we cannot do everything with our TS code, at some point there will be limitations to achieving what you want to achieve.

The scripts folder is for my bash files, bash files are plain text files that contain a series of commands to interact with the OS usually.

you may ask what i have been using Bashs in this project, I only have 2 bash scripts that

The first one is to move or copy my assets to the build folder when a build happens
The second one is to move or copy the Type folder to the build folder when a build happens so that file declarations work properly.

Test Folder

Tests are essential and important for any project regardless of what type of test you are writing.

even though I have created a package and unit tests make more sense but I have thought that integration tests will be more beneficial for my case as I need to respond back to the user.

So I have written Integration tests inside the Test Folder to test the Demo Routes and make sure that all of our supported types work properly without any errors.

Types Folder :

The types folder is a folder where my custom types lie, any TS project should have this so that your Types are not mixed with the Application logic because I can tell you, types with the code will be an easy mess very quickly.

The Rest of the Architecture
not much left to discuss, the util folder is for utility functions but we don't have any in this project to mention.

.npmignore is a file to specify which part of your project can be downloaded when someone does npm install, it's a critical file because you don't want your users to have the whole project in their node modules.
LICENSE is a file to specify which License you are using, there are many options here but mostly the open source projects go with MIT because it gives a complete right of usage of the package.
readme.md is a file you should carefully describe your package in, that's the entry point if someone sees your package and you should carefully and completely explain & document the package.

Conclusion

thanks for reading this article :)

MySQL's Data Streaming : What is it & How it works ?

Ali Amjad — Sat, 11 Jun 2022 09:27:35 +0000

In the article, I will be discussing one of the most interesting features of MySQL that you gonna need all the time and discuss how it works in the background. So let's begin 😋

I - Introduction to Data Streaming

So let's begin with What is Data? , it's information that has been translated into a form that is efficient for movement or processing. Relative to today's computers and transmission media, data is information converted into binary digital form.
Based on statistics, the amount of text data created every day across the globe is more than 18 billion. So data is being created and transferred between computers all across the globe all the time.
Data can be transferred in multiple methods, to understand the data streaming transfer, we need to understand the traditional way and learn why we need something like streaming.

A common way to transfer a file is that all the packets must arrive at the destination to be reassembled and then reach their destination, an example is when you send an image and you need all bits of this image to display it thus it must be delivered first.

However, if you are waiting for a video to load, are you waiting for the full video to download and play? Well not all of us, that's why we have streaming services like Youtube, Netflix, and others and you can start playing any of them right away, and that's where a streaming idea comes to play.

What makes streams unique, is that instead of a program reading a file into memory all at once like in the traditional way, streams read chunks of data piece by piece, processing its content without keeping it all in memory.
Instead of waiting for the 100 MB video to load, we can get it to chunk by chunk and load each 1MB as they are consumed from the stream and start displaying it right away.

II - What do we mean by Database's Data Streaming?

As we have a clear understanding of what we mean by data streaming, and how it will help us in the database world is our next topic.

Let's say we have a table from SQL that has some data in it ( roughly 1000 ) and you make a select statement to retrieve them for some O(N) calculation, it's fast enough that you don't feel any delay for your report.

However, let's say we have 1B rows ... your select statement takes enough time and your O ( N ) calculation is taking extra time as it has to wait for the rows to be retrieved and then start going through all of them to do the calculations.

Now the fun part, how can we improve this? Yes, you got it right, Let's see how streams for this select statement will help us.
Instead of waiting for 1 Billion rows, let's get a row one by one so when a row is fetched by the DB, We make some calculations on it right away as they are a chunk of this data and we process chunk by chunk, and then we send it back to the user before receiving next chunk.

By the end of this fetching, you will have a 1B calculated rows that are sent back to the user and the user didn't wait at all, and We only optimized an O ( N ) calculation, this will hugely improve your more complex calculations.

Database Streaming will reduce the user waiting time and optimize your calculations on the huge amount of data.

Note: This is the explanation for our Article only as we get deeper in the next sections, Database streaming also has a huge impact on the Data science of creating Pipelines and data lakes but that's for another article

III - Cases you can solve with MySQL's Data Streaming

Streaming your database data can have many applications. I usually use it for calculations and reports required by the business that requires calculations on Billions of rows and must be done in a fast way.

Maybe you are creating an Excel File for big data, or a PDF for a huge report then inserting one into them as they are fetched is way faster.

You may want to have a video player and store the binary data in a database and you can stream it back to the user. You may have a gallery that images fetched from DB, you can display an image by image or much more applications.

If you are a data scientist and trying to create a Pipeline by yourself to migrate data between two databases, then you can stream them daily to be in sync, Or if you are looking for a data lake to change the data then you can modify chunk by chunk while streaming the data.

In short, you can improve your current structure to be much faster for any case.

IV - Introduction to Knex.js (NPM package for Node.js)

Before coding our streamer, we will be using the Knex.js library as our query builder.

Knex.js (You know they also pronounce the "K" in the front? lol) is a "batteries included" SQL query builder for PostgreSQL, CockroachDB, MSSQL, MySQL, MariaDB, SQLite3, Better-SQLite3, Oracle, and Amazon Redshift designed to be flexible, portable, and fun to use.

They provide a beautiful way to use SQL in your Node.js, You can refer to their official documentation to know more about this amazing product.

You can check the documentation Here: Knex.js Documentation
and You can check the Streaming Documentation: Knex.js Stream Documentation

One of the Knex.js features is Streams for MySQL, You can stream your query and modify it with javascript in a very easy way.

However, Knex uses streams that are provided by original Mysql npm for node.js which is made by Felix Geisendörfer and his amazing team that made it easier for Knex to use their streaming into their library. We will discuss how the Mysql package achieved this in later sections.

V - Introduction to Streaming in Node.js

One last thing worth mentioning is the node.js Streaming module, for the implementation we will be doing in the next section, node.js streaming has also its role in it as well as the functionality from MySQL provides, so let's briefly explain what is node.js stream?

The Stream module is a native module that is shipped by default in Node.js. The Stream is an instance of the EventEmitter class, which handles events asynchronously in Node.js. Due to their superclass, streams are inherently event-based.

There are 4 types of streams in Node.js:

Writable: Used to write data sequentially
Readable: Used to read data sequentially
Duplex: Used to both read and write data sequentially
Transform: Where data can be modified when writing or reading. Take compression for an example, with a stream like this you can write compressed data and read decompressed data.

This is briefly about the node.js streaming module, for more information you can read all about streams at Node.js Official documentation.

VI - Implementation of MySQL's Data Streaming with Node.js

In this section, we will be coding our streamer with the Knex.js Package, so let's begin right away.

As a side note, Node.js and Knex.js package basic usage knowledge is required because I will be focusing on the stream only throughout the coding.

First, I will be creating a file called "stream.js" with an async function called "sample" that gonna be our sample in this article.



const database = require('./database/connection');


 async function sample(){

    console.log("Hi , This is a sample function");

 }

 sample();

we have knex mysql connection at top , and i can run this file with "node stream.js"

Then , I will be creating a table from SQL so that we can write queries on .. i will quickly write migration for it and add some test data into it.



CREATE TABLE `sample` (
  `id` int unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(255) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`)
)

I did add some dummy data into this database , around 3000 records



SELECT COUNT(*) FROM sample; //returns 3000

Now i will use Knex.js Stream function to modify each one as they are fetched



const database = require("./database/connection");

async function sample() {
  console.log("Started At :", new Date().toISOString());
  await database("sample")
    .select()
    .stream((stream) => {
      stream.on("data", (row) => {
        console.log(row.name, `At : ${new Date().toISOString()}`);
      });
    });
    console.log("Ended At :", new Date().toISOString());
}

sample();

Running this code will print the rows as they are fetched from the DB with the exact date start and end date, you can make the difference between them.

What we did, we simply write a select * for this table, and then we used the .stream function provided by knex.js, then we are listening on "data" which indicates when each row arrived, there are other events like on Error to handle the error occurrence in the stream.

This is an example of the output :

VII - How MySQL's Data Streaming works and How it's implemented with Node.js? 😱

Finally, let's discuss how this works in the background and how this stream works behind the scenes.

First, Knex.js is a Query builder that only provides a stream interface, in other meaning that they have added another layer on the top of a feature to make it easier to be used by programmers and that's why it's kinda difficult to know how it works behind the scenes from Knex.js Documentation.

Originally Stream feature comes from the original MySQL-node client package which knex.js is depending on for MySQL. the MySQL package does provide a brief on how it's working in their documentation, You can read it from the MySQL stream.

However to this point we have explained, it's unclear if it's native MySQL functionality or something made possible with node.js only.

So let's dive deeper to see how this part is coded and we could get a lead from there.

Mysql for node.js is an open-source package, so you can visit how it's made through their GitHub repository , after wandering around for some time, you will get some leads from their implementation.

if you look into this file where the code relies on , you see they have used a MySql's Text Protocol called "COM_QUERY" to make this work, so let's dive more deeply into what's this protocol do?

COM_QUERY (SELECT statement message parsing) is one of the MySQL communication Text Protocols, let's focus on what do we mean by Text Protocol and let's compare it to Binary Protocol :
The difference is really whether the protocol is oriented around data structures or text strings, for example, HTTP is a text protocol, even though when it sends a jpeg image, it just sends the raw bytes, not a text encoding of them.
So basically by Text protocols in MySQL, we can send and receive data without any encodings and the benefit of Com Query is that we can parse the text to extract our needs.
You can get a list of MySql's Communication Protocols, and a List of MySQL Text Protocols at their official Documentation.

Back to COM_QUERY, Let's get into more advanced details on how it works :

Step 1) Client Command or Client Side:

A COM_QUERY is used to send the server a text-based query that is executed immediately, in other meaning that when you provide this "Select *" and chain it to the Stream function, it will send this query to the server and then start executing right away.

Step 2) The MySQL server responds with 4 packet possibilities

1- If there is an execution error, such as SQL syntax error, return the ERR package

2- If the execution is successful, but no data is found, return the OK package

3- If the client executes load data local infile 'filename' into table

, LOCAL_INFILE_REQUEST is returned.

4- If a result set ( means there was some data ) is returned, the delivered package is Resultset.

Step 3) Let's focus on the Result Set

The Result set means the Data is Found, which means we are getting a sequence of packets.

The Result Set is the combination of two parts that we receive,

First is the column definitions, which contain information about the columns and the data types and schema details in general.
Second is the Rows, Each Row is a packet and we get the rows as packets.

The structure of the ResultSet response packet is as follows:

1 - The first packet: is the column length packet.

2 - Followed by n field description packets, each field description is a packet, After all, field information descriptions, an EOF packet or OK packet will be sent as the separator between the field definition and the data (Row).then the OK packet is returned

3 - Next is the line data packet, one data packet per line, including the packet header and the message body.

4 - The final end packet, which may also be an EOF or OK packet

In short and more generally speaking, after Com Query has been executed ... we receive a sequence of data packets that we each of them is a row.

So let's focus on the MySQL package and how they have used that, after providing a string query, they will send a com query to the server ... they receive the row packets and for each packet, they parse it to extract the row that is fetched and they use Node.js streams to emit them as events right away and you listen on "Data" to get the rows.

Well, that was a lot of theoretical explanation xD, But now we have a better understanding that it's working because of a MySql protocol.

VIII - Database Data Streaming with Other Databases and Languages

As for the Databases, I am not sure how many other databases support this but my research and playground was only MySQL.

As for other Languages than Node.js, this is only something made friendlier with the node.js streams module ... this is achievable with other languages as well, I have seen Java examples doing the same thing.

IX - Conclusion

In Article we discussed A lot About Streams, Now you should have some level of understanding of what streams are in databases. We have provided some examples and a deep explanation of what happens in the background.
Please refer back to MySQL Documentation or any links provided above to learn more about this.

X - Thank You

Thanks for reading this article, this is one of the Articles I enjoyed when I was writing it.
hopefully, you gained new information when you read through it.

You can Go through my Account for more articles.

A Logger : What to consider when creating a Logger with Node.js

Ali Amjad — Mon, 06 Jun 2022 14:28:07 +0000

Lately at work, we were facing an issue that the systems that were interacting with each other have grown into more complicated systems, it was getting difficult and more difficult to track the flow between them.

we needed more visibility of these systems which was planned to help us in areas such as :

1- Easy to Debug Issues when something fails
2- Easy to identify issues that are occurring in the run time.
3- Reducing Debugging time

So that's how we have decided on a Logger, We needed a General and Centralized Logger that we could add more visibility.

What can you log?

To be honest you can log as much as pieces of information you think are possible to have.
the way you decide for creating your own logger is to store what you need, in our case :

1 - we are keeping track of a record that was changing between multiple systems, status changes of this record and we are storing it.

2- we are storing the execution of the code, and the parameters changing between systems.

3- we are doing some heavy calculations and storing them so it will be easier to get more complicated reports from this logger

in your case :
You can track your users across your systems, track requests, Track System behavior, Track System activities, and much more.

A Logger can help you in your business decisions, you can use them in BI tools or any data science procedures.

However, there are some things to consider while creating a logger in Node.js.
You may store too much information and too much interaction with I/O with any device or with any database.

1 - Don't Block your Request-Response Cycle and Main Thread ( Node.js )

A ) Don't "Await" each one of your logger inserts, you know it will stop at each line to get success then next line?
You can parallelize them with Promise.all() and use Single await for them, the tasks will run in parallel.

Await Promise.all([promise1, promise2, promise3])

B) Let's make this better, do you know this won't get to the line until they all run in parallel and all of them succeed?

Node.js is single-threaded but it will launch other threads in the background so that they run without blocking your code, HMMMMM let's make turn this to benefit our use case but how?
in Node.js, we have to Await as well as .then to resolve promises ... so simply instead of Await and waiting for a task to complete, just add a .then() to it so that it will resolve, The Cool part is that the code execution will continue and will execute other things and this specific task runs in the background.

Promise.all([promise1, promise2, promise3]).then((values) => {
  console.log(values);
});

C ) That's really cool, now you have a nonblocking logger that runs in the background, but still the request won't end because there is something in the background that is executing, and it's taking too much time.

There is a solution for that, You need to run your loggers after the response has been sent back to the client. in Node.js, the code will continue executing even after ending the request which is basically what we want, to not make the users wait.

res.send('Hello World!')

//DO SOMETHING HERE

2- If you are storing in the database, Don't overwhelm the DB with too much inserting, Best practice is to use batch Insert or batch Update.

Or You can use Queues to reduce the load and reduce the Database CPU Usage ( if it easily hits the maximum )

Example: AWS SQS service

3- Don't overwhelm your Server with too Many HTTP Requests

if you are storing a user location, it's a bad idea to track the user for each second and let the server know to store a logger.
You can hit the Server for example per 20 seconds, or you can open other connections that HTTP like Socket Channel to send the logs and reduce the load.

4- Try To save meaningful data and try to think in the long term about where you may use these logs in a more effective way.

in the conclusion, This is how I approached this problem of Logging the things I want in an effective way that its Logging ( First Priority ) and its Not slowing down anything ( Second Priority )

SQL: One to Many Join Query in an Optimal way with Node.js

Ali Amjad — Mon, 16 May 2022 21:15:43 +0000

For the past few years, one of the queries that I needed most was making one too many queries to return a more informative response back to the client.

So what is One to Many Relationships in SQL?

Formally, A One to many relationship is a type of cardinality that refers to the relationship between two entities A and B in which an element of A may be linked to many elements of B, but a member of B is linked to only one element of A.

Along with One to Many, we Have One to One and Many to Many Relationships as well but we are on focusing on One To Many in this Article.
One to One relations are easier as only one record is connected with another one, You could retrieve the data you want in both tables with a single join.
Many to Many Relation is made of two One To Many Relationships so the Idea we are discussing here can be applied to that as well.

So What are we discussing here?
Imagine you have a User and User Hobbies Table, Each User Has Many Hobbies and Each Hobby belongs to only one user.
If you make a query over the Hobbies table then I could easily Join it with Users to retrieve the User's information related to a Hobby.
But is it helpful? a list of Hobbies? Actually in most cases No, You want a List of Users with their Hobbies and we preferably want to have an array of User Objects Each one of them should have a property which is an Array of Hobbies embedded in the user object.

Ok, but can't we just make the Join on the Users table with the Hobbies to fetch the Hobbies with the respect to the user? Again No, SQL does not work that way and it is not like NoSQL databases and does not support this kind of Query.

Hmmm using a backend language like Node.js or any other, To Find a List of Users in a separate Query and then Loop through all of them and each iteration to query on the hobbies table and embed it? This Actually Works, you will have the correct result but this is one of the worst approaches as you are hitting the Database in a Loop.

So let's do this in an Optimal Way that does not hurt our DB and we are not making too many connections to the DB and yet we achieve the same output.

First, Am gonna Create these two tables with the below schema on SQL , they are in the simplest form.



CREATE TABLE `user` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`id`)
);

CREATE TABLE `user_hobbies` (
  `id` int NOT NULL AUTO_INCREMENT,
  `user_id` int NOT NULL DEFAULT '0',
  `hobby_name` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`id`)
);

Now Tables are Created, I will put some Dummy data into them so that we will have something to work with.

You may have noticed that I am adding User_id without having SQL relationships, its works that way, and it's preferable and we will discuss why in another article.

ok let's start coding

init new Node.js Project
I will be using two packages
1. Knex.js as Query Builder
2. Lodash for Computation
Start

First I will be creating a Function and Fetch All of the Users, Please refer to Knex.js Documentation to know how to make connections and start querying.



const _ = require('lodash');
const lodash = require('lodash');
const database = require('/database/connection'); //DB Connection

function getAllUsersWithHobbies() {}

getAllUsersWithHobbies()

Lets Fetch All of the Users first.



async function getAllUsersWithHobbies() {
    const users = await database("user").select();    
}

Now you have an Array of Users , So what should you do next is to extract their Ids.



async function getAllUsersWithHobbies() {
    const users = await database("user").select();  //[ { id: 1, name: 'Ali' } ]
    const usersIds = _.map(users, (el) => el.id);  //[ 1 ]
}

Perfect , now we make a query on the Hobbies to find all the Hobbies that their user_id is in our UserIds variable.



async function getAllUsersWithHobbies() {
    const users = await database("user").select();  
    const usersIds = _.map(users, (el) => el.id);  
    const Hobbies = await database('user_hobbies').select().whereIn('user_id', usersIds) //returns 2
}

The next step is to Group The Hobbies By User_id , using GroupBy Function from Lodash ... it will return an Object that has user_id as keys and Array of the Hobbies belongs to a User as value.




async function getAllUsersWithHobbies() {
    const users = await database("user").select();  
    const usersIds = _.map(users, (el) => el.id);  
    const Hobbies = await database('user_hobbies').select().whereIn('user_id', usersIds) 
    const groupedHobbies = _.groupBy(Hobbies, 'user_id');

    // groupedHobbies: {
    //     '1': [
    //        { id: 1, user_id: 1, hobby_name: 'First Hobby' },
    //        { id: 2, user_id: 1, hobby_name: 'Second Hobby' }
    //     ]
    //   }
}

Finally We loop through our users and we can point to the Hobbies object with the user.id



async function getAllUsersWithHobbies() {
    const users = await database("user").select();  
    const usersIds = _.map(users, (el) => el.id);  
    const Hobbies = await database('user_hobbies').select().whereIn('user_id', usersIds) 
    const groupedHobbies = _.groupBy(Hobbies, 'user_id');

    const usersEmbedded = _.map(users, (record) => {
            return {
                ...record,
                hobbies: groupedHobbies[record.id] ,
            };
        }); 

       return usersEmbedded;
}

an Example of final output :

Well, we have achieved the exact output we have wanted by hitting the database only twice to retrieve all the hobbies for all the users and embed them as well.

This is a simple technique yet so powerful that you want to do all of your embedded this way.

Thanks for reading this Article

Simplest Uploader? Creating Powerful Node.js Object Uploader

Ali Amjad — Thu, 12 May 2022 19:21:21 +0000

Uploading Objects to the server is one of the Key Concepts of backend development and web development in general.

it's quite rare to see a website without images, most of the time these websites are not static and the images, as well as details, are managed from an Admin Panel Dashboard, Or you have seen Forms in websites to attach a file with the details.

An Uploader is your way on how to let the server handle the upcoming files from the client-side.

in this article, we will discuss creating a file uploader that is not limited to a type of file but rather works for almost all of the file types and without using any extra package.

Nowadays approaches for creating an Uploader versus what we are building

There are two common methods to send a file to the server ( the whole file, so no streams here )

Sending File as Form-data: For example, submitting an HTML form with attached files to it will be labeled as Multipart.
This usually requires a body parser that can parse this complex data from the body such as Multer.
Sending File as String: Such as converting the File to base64 encoding and sending it as JSON in the Body.
one thing you should be aware of, Base64 encoding writes each 3 bits as 4 bits which increases the file size by up to %30.

What we are doing :

We won't be using any parser for the multi-part to keep it simple, thus we accept the base64 or image text area.
even though the size increases, you shouldn't use this method for very large files and it doesn't make a lot of difference in small files(up to 100MB).
we only use functions of Node.js's Fs modules

There are a lot of packages out there, but sometimes your needs are just simple things to accept anything with no restrictions.

So let's begin

any file that hits this API should be encoded with base64, you can find many libraries to have base64 uploader in Front-End

I have got this 1px by 1px image, just to not get the string too large.

I have converted it to base64 from an online converter Base64 encoder

Let's start Coding

Creating new node.js project =>

First I will run this to create a node.js project

npm init

Then I will create index.js as our playground for our uploader.

At First, I will import Node.js's Fs core module and put our base64 example in the file.

const fs = require("fs").promises;

let exampleImage = "data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAAAXNSR0IArs4c6QAAAAxJREFUGFdjcOyfCQACfgFqcbSa6QAAAABJRU5ErkJggg=="

Then we will add our function upload, simply we will have a name and which I hardcoded the file suffix for simplicity but usually, base64 encoded string also includes the file type and you can put it as name.${extract the file suffix}.


const upload = async (base64) => {
  const fileSuffix = "png"
  const name = `anyFileName.${fileSuffix}`
  const dir = `${__dirname}/file/${name}`;
  const base64Data = Buffer.from(
        base64.replace(/^data:image\/\w+;base64,/, ""),
        "base64"
      ); 
  await fs.writeFile(`${dir}`, base64Data);
};

then we will have the directory we want to save and basically extract the base64 from the Base64Data and turn it into a buffer.

lastly we write the data to the specified folder .

then run the function

upload(exampleImage);

this is the result of running the program with

node index.js

the output is that we have successfully saved an image.

With only this, you can save any object you want .. you need the base64 and simply what that file is that is uploaded. if it's a PDF or Image or any file.

Thanks for reading this article.

Data Migration Between Two MongoDB Atlas Instances ( Connections )

Ali Amjad — Mon, 09 May 2022 21:33:33 +0000

As for how easy it can be to set up a MongoDB Atlas account and database, it will just become harder when you try to move the data around.

Lately, at our company, we have decided to migrate our current development MongoDB Instance into another Instance and In this article, I will discuss how I approached and solved this task.

So the challenging part of this Migration was our database design, because of its complexity, the Application is a built upon Microservices and Microservice Architecture that includes around +30 Microservices with +10 Databases that handle Different Groups of Microservices separately, and then each of them contains around hundreds of collections that even each of them contains thousands of data.

According to the Official Documentation of MongoDb, you could export each collection separately which is not helpful for our case because we are dealing with a lot of collections.

Another Solution might be using the MongoDump command to dump the whole Database together, but again not that helpful as it's done manually for exporting and importing each of the Databases.

After enough research regarding the best way to handle this situation, I have come to a tool that could do this in a matter of seconds which I will be discussing below.

First of All, Prepare your Host and Target MongoDB URI or at least a connection for each of them
Then, We are using a tool called NoSQL Booster.

NoSQLBooster is a cross-platform GUI tool for MongoDB v2.6-5.0, which provides a build-in MongoDB script debuggerMongoDB script debugger, and comprehensive server monitoring tools, chaining fluent query, SQL query, query code generator, task scheduling, and much more.

After you have installed the tool, you will be popped up with adding connections from the first step.

in this step , I am Connecting via my Atlas URI, so you can simply connect other atlases or any MongoDB from any server.

you will see your connections loaded in the side like mine in the picture

Only part left is pretty simple, you copy a database and simply paste it into the other connection. it is simple as that, It will load the DB with all of its data and indexes into the other one.

A) Copy A Database from the Host Connection

B) Paste A Database to the Target Connection

in my case, I did the copying operation around +10 times and our new database is good to go :).

Thanks for reading this Article, Just wanted to share this experience I had today in this article.

can you do all the DBs together with one click? I am not sure, maybe there is a better solution but this is quickest i have found and used. :)