DEV Community: Antonello Zanini

CapMonster Cloud Review: The All-in-One CAPTCHA Solving Service

Antonello Zanini — Wed, 08 Oct 2025 09:58:56 +0000

Raise your hand if you think CAPTCHAs are annoying. Well, you’re not alone… For human users, CAPTCHA challenges are frustrating, but for automation workflows (e.g., web scraping bots or AI agents), they represent a serious roadblock!

That’s why, over the past few years, several CAPTCHA-solving services have popped up. They all promise to solve CAPTCHAs programmatically, but can they really deliver? Today, I’m reviewing CapMonster Cloud, one of the major players in the CAPTCHA-solving industry, to see if it lives up to the hype.

Follow me on this journey as I explain what the company is, the services they provide, how it all works with practical examples, their pricing, and—of course—a final verdict!

What Is CapMonster Cloud?

CapMonster Cloud is an online service that leverages artificial intelligence to quickly solve various types of CAPTCHAs in the cloud. Its solutions are designed for anyone who needs to bypass CAPTCHA challenges during web scraping, automation, or testing workflows.

This service is developed by ZennoLab, a company that provides a comprehensive suite of automation solutions.

CapMonster Cloud is trusted by over 1,000 businesses worldwide and has received excellent user feedback, including a 4.8/5 rating on Trustpilot, 5/5 on Slashdot, and 4.8/5 on SourceForge based on more than 120 verified reviews.

Main Features

These are the main features, capabilities, and aspects that characterize CapMonster Cloud:

AI-powered cloud-based CAPTCHA solving for automation tasks.
Reliable and fast API designed for high scalability.
Browser extensions for Chrome and Firefox for automatic CAPTCHA solving.
Ready-to-use SDKs in Python, JavaScript, C#, Go, and PHP.
Support for integration with external services.
Support for a wide range of CAPTCHA types, including reCAPTCHA v2/v3, Cloudflare Turnstile, DataDome, Tencent CAPTCHA, Amazon WAF, GeeTest, Imperva, Text CAPTCHAs, and others.
Ethical and transparent CAPTCHA automation principles.
Affiliate program available.
Fast and responsive customer support, available even on Telegram.

Product Lineup: CAPTCHA-Solving Solutions

Let me introduce the products offered by CapMonster Cloud API. Here, I’ll analyze how they work, the experience they provide, and their pros and cons.

For a quick overview, take a look at the summary table below:

Feature	CapMonster Cloud API	CapMonster Cloud Browser Extensions
Purpose	Cloud-based CAPTCHA-solving service for automation and development	Browser-based tool to automatically solve CAPTCHAs while browsing
Platform	Any tool or programming language that can make HTTP requests	Chrome and Firefox
Setup	Call API endpoints authenticated with your CapMonster Cloud API key to send CAPTCHA data and receive the solution	Install the extension, enter your API key, and let the extension solve the CAPTCHA for you

What I can say before diving into the details is that an API service and browser extensions are the two products you’d expect from a top-tier CAPTCHA solver. On top of that, as you’re about to discover, each comes with some unique perks that make them stand out.

Time to learn more!

CapMonster Cloud API

CapMonster Cloud API is a cloud CAPTCHA-solving service that automates CAPTCHA recognition and bypass. It exposes several HTTP endpoints so that you can send CAPTCHA data to the cloud and receive a solution in response. That’s ideal for developers seeking a programming language–agnostic CAPTCHA-solving solution.

In particular, the main endpoints exposed by CapMonster Cloud are:

POST /createTask: Create a CAPTCHA solving task. Discover all supported CAPTCHA types.
POST /getTaskResult: To poll the task and retrieve the solution to the CAPTCHA.
POST /getBalance: To check your account balance.
GET /getUserAgent: Returns the latest User-Agent string for Chrome on Windows. Setting a valid User-Agent helps the cloud solver better emulate a real browser and reduces blocks for certain CAPTCHA types.

Note: /getBalance and /getUserAgent are there only for account management and improving accuracy, respectively. While not strictly required, they’re a nice-to-have addition!

Given the available endpoints, this is how the CAPTCHA-solving process works:

Create a task: Call /createTask with your Cloud Monster API key, the task type for your CAPTCHA type, and CAPTCHA details (e.g., key, URL, etc.). This API returns a taskId.
Poll for a result: Poll /getTaskResult with your CapMonster Cloud API key. When the CAPTCHA-solving task finishes, the API will return a "ready" status and an object containing the solution for your task type (e.g., the text answer, a gRecaptchaResponse token, etc.).
Use the solution: After receiving the solution, use it to solve the CAPTCHA or submit the token to validate the target site’s form, API call, page loading, etc.

As you can see, the process is pretty simple and similar to other CAPTCHA-solving services. So, nothing new here!

⚙️ Special features:

Extra endpoints for account management (/getBalance).
Public API (/getUserAgent) to fetch the User-Agent header set by the latest version of Chrome on Windows.

👍 Pros:

Supports ~20 different CAPTCHA types, each clearly documented and with snippets in C#, Python, JS, Go, and PHP.
Intuitive API-based workflow using only two main endpoints: /createTask and /getTaskResult.
Cloud-based, requiring no software installation, and usable from any programming language that can make HTTP requests.
Support for custom proxy integration.
Clear error descriptions, with technical support available via a support form, the Telegram channel, or live chat on the website.
High-level SDKs in Python, JavaScript, C#, Go, and PHP, so you don’t have to call the APIs directly.

👎 Cons:

Temporary bans (usually ~10 minutes) occur for repeated errors, such as requests without a key or with an invalid key, requests on a zero account balance, exceeding the task check limit (~120 per task), etc.

Browser Extensions

CapMonster Cloud also comes with dedicated browser extensions, which enable you to automatically solve CAPTCHAs encountered while browsing. These are available for Chrome and Firefox, and they connect to your CapMonster Cloud account via an API key.

These extensions are perfect for professionals who want to save time and boost productivity by automating repetitive CAPTCHA challenges. Below is how they work:

Install the extension in Chrome or Firefox.
Enter your CapMonster Cloud API key and ensure your account balance is not zero.
The extension automatically detects CAPTCHAs, sends them to the cloud, and fills in the solution in the background.

Quite easy, isn’t it!?

Also, they support developers who want to integrate them into browser automation tools like Selenium, Playwright, or Puppeteer. That's possible because the extensions can be downloaded as a local archive, allowing configuration and programmatic integration into browser instances controlled by automation scripts.

The two browser extensions also add a dedicated “CapMonster Cloud” tab in your browser’s DevTools. This tab displays details for the CAPTCHAs on the current web page, providing all the information you need to populate the body for /createTask requests:

As of this writing, the CapMonster Cloud extension for Chrome has over 20k downloads and a 4.3/5 rating based on 73 reviews.

⚙️ Special features:

Simplifies CAPTCHA recognition for API-based solving via a “CapMonster Cloud” tab in the DevTools.
Supports integration with browser automation tools like Playwright, Selenium, and Puppeteer.
Option for automatic clicking after CAPTCHA completion.
Adds contextual menu options for solving text CAPTCHAs on the fly.
Sends browser events that can be used for advanced workflows.

👍 Pros:

Automates CAPTCHA solving, saving you time.
Works in the background without any user intervention.
Available in both Chrome and Firefox.

👎 Cons:

May have known issues detecting CAPTCHA keys hidden inside shadow DOMs in closed mode.

Hands-On: Solving a CAPTCHA with CapMonster Cloud

You know what CapMonster Cloud has to offer, but at the end of the day, what really matters is the developer experience and whether the CAPTCHA-solving solution actually works.

So, let’s not waste any more time and see CapMonster Cloud in action with a real-world CAPTCHA-solving scenario, both via the API and the browser extensions!

Before Setting Started

To follow along, make sure you have:

Python installed locally.
The CapMonster Cloud extension installed in your browser.
A CapMonster API key.

To get your CapMonster API key, sign up or log in, go to your dashboard, add some funds, and copy your API key:

Store your API key in a safe place, because you’ll need it to integrate with CapMonster Cloud in both the API and browser extension walkthroughs.

Specifically, I’ll show you how to solve the CAPTCHA on Google’s reCAPTCHA demo page. There, a reCAPTCHA v2 “I’m not a robot” checkbox prevents forms from being submitted by bots. Once the CAPTCHA is solved and the form is submitted correctly, you’ll see the “Verification Success… Hooray!” message:

For those not familiar with this challenge, the “I’m not a robot” checkbox is the most common version of reCAPTCHA v2. This is how it works:

If the user is deemed human, the checkbox displays a green checkmark, allowing them to proceed without further interaction.
If suspicious activity is detected, a visual challenge is presented (e.g., selecting all images containing a specific object, like a bus or traffic light).

reCAPTCHA v2 is one of the most widely used CAPTCHA types, typically associated with form submissions and anti-bot detection. Even Google itself uses it in its search pages when it suspects bot activity:

Let’s verify if CapMonster Cloud can handle it!

With CapMonster Cloud API

Once you know you’re dealing with a reCAPTCHA v2, start by taking a look at the official documentation.

💡 Pro tip: If you’re unsure which type of CAPTCHA you’re dealing with, you can check the info in the “CapMonster Cloud” tab from the browser extension, like I showed earlier.

For this example, the reference is the RecaptchaV2Task documentation page:

The documentation is thorough and includes all the details you need to solve this type of CAPTCHA. Don’t worry, though. If you need more guidance, the CapMonster Cloud blog has step-by-step guides for all supported CAPTCHA types. For example, take a look at these two blog posts:

Solving a reCAPTCHA v2: A comprehensive guide for web scrapers
Everything about CAPTCHA recognition Long story short, the CapMonster Cloud team has really got you covered!

For reCAPTCHA v2, the required body for the /createTask API endpoint looks like this:

    {
      "clientKey":"<YOUR_CAPMONSTER_CLOUD_API_KEY>",
      "task": {
            "type":"RecaptchaV2Task",
            "websiteURL":"<WEB_PAGE_URL>",
            "websiteKey":"<PUBLIC_RECAPTCHA_KEY>"
      }
    }

In other words, you need to retrieve the websiteKey from the target page. This is the site’s API key used to load the CAPTCHA and is embedded directly in the page. You can find that by exploring the DOM (e.g., search for "key" or similar):

Or, to save time, you can use the “CapMonster Cloud” tab in your browser’s DevTools, as shown earlier. It’s always nice to have multiple options!

In this case, the body for the /createTask API call looks like this:

{
  "clientKey":"<YOUR_CAPMONSTER_CLOUD_API_KEY>",
  "task": {
        "type":"RecaptchaV2Task",
        "websiteURL":"https://www.google.com/recaptcha/api2/demo",
        "websiteKey":"6Le-wvkSAAAAAPBMRTvw0Q4Muexq9bi0DJwx_mJ-"
  }
}

That’s literally all the info you need to solve a reCAPTCHA v2 using CapMonster Cloud. Pretty simple, right?

Now, get ready to verify that the CAPTCHA-solving functionality works. For a simplified setup, I’ll use the official CapMonster Cloud Python SDK.

First, install it with:

pip install capmonstercloudclient

Then, you can utilize it to solve a CAPTCHA with just a few lines of code:

import asyncio
from capmonstercloudclient import CapMonsterClient, ClientOptions
from capmonstercloudclient.requests import RecaptchaV2Request

# Replace it with your CapMonster Cloud API key or read it from the envs
CAPMONSTER_CLOUD_API_KEY = "<YOUR_CAPMONSTER_CLOUD_API_KEY>"

async def main():
    # Initialize a CapMonster Cloud API client
    capmonster_client_options = ClientOptions(api_key=CAPMONSTER_CLOUD_API_KEY)
    capmonster_client = CapMonsterClient(options=capmonster_client_options)

    # Request CAPTCHA solving from the cloud
    request = RecaptchaV2Request(
        websiteUrl="https://www.google.com/recaptcha/api2/demo",
        websiteKey="6Le-wvkSAAAAAPBMRTvw0Q4Muexq9bi0DJwx_mJ-"
    )
    response = await capmonster_client.solve_captcha(request)
    print(response)

if __name__ == "__main__":
    asyncio.run(main())

Behind the scenes, solve_captcha() does all the heavy lifting for you:

Creates a RecaptchaV2Task with the given CAPTCHA details by calling the createTask API endpoint.
Polls the /getTaskResult endpoint until the solution is ready.
Returns the final token.

That's precisely the same API-based CAPTCHA solving flow we discussed earlier.

response will look something like this:

{
  "gRecaptchaResponse": "0cAFcWeA6nLpY..."
}

gRecaptchaResponse is the token you get after solving a reCAPTCHA.

To complete the flow, you need to programmatically simulate the form submission on the target page, like so:

# pip install requests
import requests

# CAPTCHA solving code...

# Exract the solutuon token from the response 
g_recaptcha_response = response.get("gRecaptchaResponse")

# Simulate the form submission on the target page
page = requests.post(
    "https://www.google.com/recaptcha/api2/demo",
    headers={
        "Content-Type": "application/x-www-form-urlencoded",
    },
    data={
        "g-recaptcha-response": g_recaptcha_response
    },
)

# Retrieve the HTML document from the response and print it
html = page.text
print(html)

The result will be the HTML of the verification success page, confirming that the CAPTCHA was solved correctly (note the “Verification Success… Hooray!” message):

Mission complete! CAPTCHA solved, all with just a handful of lines of code.

With Browser Extensions

Open the CapMonster Cloud extension, make sure it’s set to “On”, paste your API key into the designated field, and hit the save button:

Your account balance will also be displayed in the extension UI, so you can always check it at a glance:

That’s a nice touch since you don’t want to run out of credits mid-task. This simple feature lets you keep an eye on them.

Next, make sure “ReCaptcha2” solving is enabled, set to “Click”, and that the “Auto solve” and “Auto click” options are turned on:

No need to go into detail about these options, as they’re pretty straightforward.

Visit the target page, and you’ll see the CAPTCHA get automatically checked (note that the mouse pointer doesn’t move):

That wasn’t magic—it’s just the CapMonster Cloud extension doing its job!

Now, press the “Submit” button, and you’ll see that the CAPTCHA has been solved successfully:

If you don’t want to click “Submit” manually, you can right-click, access the context menu, and select the “Select element for submit” option:

This way, the extension will detect the button and handle it automatically, saving you an extra click.

Amazing! There’s not much to comment here, as the experience is as effortless as you could hope for...

➡️ Want to bring this seamless CAPTCHA-solving experience into your automated browser scripts? Check out the official guide: “Integrating the Extension into Selenium using Node.js.”

Pricing Breakdown and Competitor Comparison

CloudMonster Cloud charges per CAPTCHA solved, following this pricing table:

CAPTCHA Type	Pricing
reCAPTCHA v2	$0.04 per 1k images, $0.60 per 1k tokens
reCAPTCHA v3	$0.90 per 1k tokens
reCAPTCHA Enterprise	$0.04 per 1k images, $1.00 per 1k tokens
GeeTest	$1.20 per 1k tokens
Cloudflare Turnstile	$1.30 per 1k tokens
Cloudflare Bot Challenge	$1.30 per 1k tokens
DataDome	$2.20 per 1k tokens
Tencent CAPTCHA	$1.60 per 1k tokens
Amazon WAF	$1.40 per 1k tokens
Faucet Pay	$1.00 per 1k tokens
Imperva (Incapsula)	$2.00 per 1k tokens
Prosopo	$1.30 per 1k tokens
Text CAPTCHA	$0.30 per 1k tokens
Temu	$2.00 per 1k tokens
Yidun	$1.00 per 1k tokens
MTCaptcha	$1.50 per 1k tokens
Altcha	$0.80 per 1k tokens

🚨 Important: Proxy traffic used by the Cloud solver for non-proxyless CAPTCHA tasks is included in the above prices.

In other words, to make things simple, every time you use CloudMonster Cloud to solve a reCAPTCHA v2, you pay $0.0006 per solution token (equivalent to $0.60 per 1,000 tokens). Put differently, $1 covers roughly 1,666 reCAPTCHA v2 challenges.

ℹ️ Note: Payment options are flexible and include PayPal, MasterCard, Visa, Wire transfers, cryptocurrencies, UnionPay, and MIR (payments in Russian rubles). So there’s really an option for everyone.

Based on my experience in the web scraping industry, these prices are competitive—among the best compared to other top CAPTCHA-solving providers. Considering that I achieved a 99% success rate in my tests, I believe CloudMonster Cloud is well worth the money you’re paying!

Final Verdict

In this review, I had the chance to try CapMonster Cloud in depth and see for myself whether it really is one of the best CAPTCHA-solving solutions available on the web.

The experience was fantastic. That's true for both developers using their API and SDKs, and regular users leveraging the browser extension to save time on CAPTCHA-heavy tasks. In both cases, the underlying AI-solving system doesn’t disappoint.

I’ve personally seen it work its magic a few times—like automatically selecting all the traffic lights in a reCAPTCHA challenge—saving a lot of manual effort. I imagine the same magic is applied in the cloud when solving CAPTCHAs via API

There are a few extra features worth noting, too. The CAPTCHA recognition tool available in the extension is a huge time-saver for developers. I also appreciated that the extensions are provided as downloadable archives, making it easy to integrate into browser automation setups.

Considering the competitive pricing, pay-for-success model, the long list of features, and the extensive documentation and blog posts, I can confidently say that CapMonster Cloud is one of the best CAPTCHA-solving services out there—and hands down the best I’ve personally tried!

Review score: 9.6/10

Advanced JavaScript Asynchronous With Async.js

Antonello Zanini — Sat, 07 Jun 2025 12:11:10 +0000

Asynchronous JavaScript has changed the way we approach backend and frontend development. Still, writing asynchronous code remains complex, especially considering that many native JavaScript functions do not work well with async operations. This is exactly where the Async.js library comes in!

As a set of powerful utility functions, Async provides everything you need to handle complex JavaScript async scenarios in just a few lines of code. Given how common these challenges are, it is no surprise that the library has over 50 million weekly downloads!

Let’s learn why asynchronous programming is challenging in JavaScript, how Async.js aims to resolve these issues, what features the library offers, and how to use it in two real-world examples.

JavaScript Asynchronous Development: Benefits and Challenges

The introduction of Promises and the async and await constructs in JavaScript has revolutionized web development. Handling asynchronous operations is simpler than ever, as you can now write async code that looks and reads like synchronous code. For more information, take a look at our article on JavaScript asynchronous programming tips, tricks, and gotchas.

Among the main benefits of using Promises along with async/await are improved code readability and simpler error handling through try...catch blocks. However, challenges do exist—such as managing complex asynchronous flows or iterating over async tasks.

Sure, some of those issues can be mitigated with best practices for async programming in JavaScript, such as reducing the callback hell. Yet, the best solution might be to use a dedicated library like Async, which provides dozens of functions to streamline async logic.

Learn more about Async in the chapter below!

Async.js: A Library For Simplified Asynchronous JavaScript

Async, also known as Async.js, is a utility module that provides powerful functions for working with asynchronous JavaScript. It supports Node.js and the browser, for simplified asynchronous programming in both the backend and the frontend.

Async includes around 70 functions for dealing with asynchronous collections, control flow, and other useful operations—such as logging and memorization. The GitHub repository of the project boasts over 28k stars, and the npm package has almost 55 million weekly downlands.

You can install Async via the async package with the command below:

npm install async

Or, if you want the pure ESM version of Async, install async-es:

npm install async-es

If you are a CommonJS user, you can now import async in your code with:

const async = require("async");

Or, if you are an ESM user, import it as below:

import async from "async";

Functions Offered By Async.js

Time to explore the most useful methods provided by Async, categorized as in the official documentation in:

Collections: Functions for manipulating collections, such as arrays and objects.
Control Flow: Functions for controlling the flow through a script.
Utilities: Utility functions for JavaScript async development.

Note that each Async.js function accepts an optional callback as the last argument. If provided, the callback is executed when the function completes or if an error occurs. If the callback is omitted, the Async function returns a Promise.

For example, you can use the Async map() function with a callback as follows:

const async = require("async");

async.map(
  [1, 2, 3],
  (num, callback) => {
    setTimeout(() => callback(null, num * 2), 100);
  },
  (err, results) => {
    if (err) {
      console.error(err);
    } else {
      console.log(results); // output: [2, 4, 6]
    }
  },
);

map() from Async.js concurrently applies an async transformation function to each item in a collection. It processes each item in parallel and returns a new array of results, preserving the original order of the items.

Or, equivalently, without the callback:

import async from "async";

async
  .map([1, 2, 3], (num) => {
    return new Promise((resolve) => {
      setTimeout(() => resolve(num * 2), 100);
    });
  })
  .then((results) => {
    console.log(results); // output: [2, 4, 6]
  })
  .catch((err) => {
    console.error(err);
  });

The above code can also be re-written to use the asnyc/await syntax:

import async from "async";

(async () => {
  try {
    const results = await async.map([1, 2, 3], async (num) => {
      return new Promise((resolve) => {
        setTimeout(() => resolve(num * 2), 100);
      });
    });
    console.log(results); // output: [2, 4, 6]
  } catch (err) {
    console.error(err);
  }
})();

Collections

Async provides more than 35 functions for handling JavaScript collections concurrently. Here, we will focus on the 5 most popular ones, but you can find them all in the documentation.

Note that for most functions in this category, there are two additional special versions:

functionNameLimit(): Accepts a limit argument, and runs a maximum of limit async operations at a time.
functionNameSeries(): Runs only a single async operation at a time to respect the order of the collection.

For instance, map() has the mapLimit() and mapSeries() versions.

detect

detect() returns the first item in a collection that meets an asynchronous condition. The test function is applied to the collection in parallel, so the result may not be the first item from left to right. If order matters, use detectSeries().

Example:

const numbers = [1, 2, 3, 4, 5];

detect(
  numbers,
  (num, callback) => {
    // sample async test to run on the current item
    setTimeout(() => {
      callback(null, num % 2 === 0);
    }, 100);
  },
  (err, result) => {
    if (err) {
      console.error(err);
    } else {
      console.log(result); // potential output: 2 (or 4)
    }
  },
);

each

each() simultaneously applies a given function to each item in a collection. If any of the function calls return an error, the main callback is immediately triggered with that error.

Example:

import each from "async/each";

const items = [1, 2, 3, 4];

each(
  items,
  (item, callback) => {
    // sample async operation to execute on the current item
    setTimeout(() => {
      console.log(item * 2);
      callback();
    }, 100);
  },
  (err) => {
    if (err) {
      console.error(err);
    } else {
      console.log("All items processed");
    }
  },
);

every

every() checks if all items in a collection meet an asynchronous condition. If any of the items fail the condition, it immediately calls the main callback with false.

import every from "async/every";

const numbers = [2, 4, 6, 8];

every(
  numbers,
  (num, callback) => {
    // sample async test to run on the current item
    setTimeout(() => {
      callback(null, num % 2 === 0);
    }, 100);
  },
  (err, result) => {
    if (err) {
      console.error(err);
    } else {
      console.log(result); // output: true (as all numbers are even)
    }
  },
);

filter

filter() returns a new array with items from the collection that pass an asynchronous truth test, maintaining the original order.

Example:

import filter from "async/filter";

const numbers = [1, 2, 3, 4, 5, 6];

filter(
  numbers,
  (num, callback) => {
    // sample async test to run on the current item
    setTimeout(() => {
      callback(null, num % 2 === 0);
    }, 100);
  },
  (err, results) => {
    if (err) {
      console.error(err);
    } else {
      console.log(results); // output: [2, 4, 6]
    }
  },
);

reduce

reduce() combines items in a collection into a single value by applying an async function to each item, one at a time, using a given starting value.

Example:

import reduce from "async/reduce";

const numbers = [1, 2, 3, 4];

reduce(
  numbers,
  0,
  (acc, num, callback) => {
    // sample async function to combine items with
    setTimeout(() => {
      callback(null, acc + num);
    }, 100);
  },
  (err, result) => {
    if (err) {
      console.error(err);
    } else {
      console.log(result); // output: 10 (1 + 2 + 3 + 4)
    }
  },
);

Control Flow

Async offers over 25 functions for simplifying asynchronous control flow. Below, we will focus on the 3 functions. Explore them all in the documentation.

parallel

parallel() runs multiple asynchronous tasks at the same time. Each task starts immediately without waiting for others to complete. If any task returns an error, the main callback is triggered with that error. When all tasks finish, the results are returned in an array (or an object if an object is used for tasks).

Example:

import parallel from "async/parallel";

parallel(
  // sample parallel tasks
  [
    (callback) => setTimeout(() => callback(null, "Task 1"), 200),
    (callback) => setTimeout(() => callback(null, "Task 2"), 100),
  ],
  (err, results) => {
    if (err) {
      console.error(err);
    } else {
      console.log(results); // output: ["Task 1", "Task 2"]
    }
  },
);

queue

queue() creates a queue to handle tasks with a set concurrency limit. Tasks are processed in parallel up to the limit. New tasks wait in the queue until a worker becomes available. When each task completes, the callback is called.

Example:

import queue from "async/queue";

const taskQueue = queue((task, callback) => {
  console.log(`Processing ${task.name}`);
  // sample async task
  setTimeout(callback, 100);
}, 2); // limit concurrency to 2

// loading all tasks into the queue
taskQueue.push([{ name: "Task 1" }, { name: "Task 2" }, { name: "Task 3" }]);

// executing the tasks in the queue
taskQueue.drain(() => {
  console.log("All tasks completed");
});

series

series runs tasks one after the other, waiting for each to complete before starting the next. If a task fails, no further tasks are run, and the main callback is called with the error. When all tasks are complete, the results are returned in an array (or an object if an object is used).

Example:

import series from "async/series";

series(
  // sample async tasks to run sequentially
  [
    (callback) => setTimeout(() => callback(null, "First Task"), 200),
    (callback) => setTimeout(() => callback(null, "Second Task"), 100),
  ],
  (err, results) => {
    if (err) {
      console.error(err);
    } else {
      console.log(results); // output: ["First Task", "Second Task"]
    }
  },
);

Utilities

Async offers a few utility functions to make it easier to handle asynchronous tasks. We will analyze only 3 of them, but you can find them all in the documentation.

log

log() prints the result of an async function directly to the console. If the async function returns multiple values, each value is logged in order. This function only works in Node.js and in frontend applications that support console.log() and console.error().

Example:

import log from "async/log";

// sample async task
function asyncTask(callback) {
  setTimeout(() => callback(null, "Result 1", "Result 2"), 100);
}

log(asyncTask); 
// output in the console: 
// "Result 1" 
// "Result 2"

memoize

memoize() caches the result of an async function. This way, repeated calls with the same arguments use the cached result instead of running the function again. An optional hash function can be provided to customize how arguments are used as keys in the cache.

If you are not familiar with memoization, read our article on memoizating functions for performance.

Example:

import memoize from "async/memoize";

// sample async function
const slowFunction = (num, callback) => {
  setTimeout(() => callback(null, num * 2), 100);
};

const memoizedFunction = memoize(slowFunction);

memoizedFunction(5, console.log); // takes 100ms, logs "10"
memoizedFunction(5, console.log); // cached, logs "10" immediately

timeout

timeout() sets a maximum time limit on an async function. If the function does not finish within the specified time, it returns a ETIMEDOUT error instead.

Example:

import timeout from "async/timeout";

// sample async task
function longTask(callback) {
  setTimeout(() => callback(null, "Done!"), 200);
}

// apply the timeout function
const limitedTask = timeout(longTask, 100);

limitedTask((err, result) => {
  if (err) {
    console.error(err.message); // output: "Callback function timed out."
  } else {
    console.log(result);
  }
});

Async.js in Action

Now that you have an idea of what Async.js has to offer, you are ready to see it in action in real-world scenarios!

Async Loop Functions

Functions like forEach(), reduce(), map(), and filter() are among the most common in JavaScript development. Still, challenges arise when using these methods with asynchronous calls or promises. In this case, results may not be what you would expect. For more information, follow the 4-part guide on async loops, and why they fail.

For example, see this async function to simulate an API call:

function asyncCall(timeToWait, dataToReturn) {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(dataToReturn);
    }, timeToWait);
  });
}

This returns a Promise that resolves after a specified delay, just like what would happen when calling an endpoint.

Now, use forEach() to call the above function on each element in an array:

const values = [1, 2, 3, 5, 8];

console.log("START using forEach");
values.forEach(async (value) => {
  const data = await asyncCall(1000 * value, `data #${value}`);
  console.log(data);
});
console.log("END using forEach");

Run the code and the output will be:

START using forEach
END using forEach
data #1
data #2
data #3
data #5
data #8

As you can see, forEach() ends immediately, while the results from asyncCall() are logged only afterward. This occurs because forEach() does not wait for the Promises to resolve. In particular, it launches all asynchronous tasks, but it returns immediately. Thus, the async logic defined in the async callbacks passed to forEach() executes only after forEach() has returned.

To properly handle asynchronous calls in an iteration, you can use Promises with a custom asyncForEach function:

const asyncForEach = (array, callback) => {
  return array.reduce((promise, item) => {
    return promise.then(() => callback(item));
  }, Promise.resolve());
};

console.log("START using asyncForEach");
asyncForEach(values, async (value) => {
  const data = await asyncCall(1000 * value, `data #${value}`);
  console.log(data);
}).then(() => console.log("END using asyncForEach"));

Thanks to Promise chaining with array.reduce(), you can control the flow of execution and ensure that each async operation completes in series.

With this approach, the output will be:

START using asyncForEach
data #1
data #2
data #3
data #5
data #8
END using asyncForEach

While that solution works as desired, it introduces complexity and boilerplate code. This is where the Async.js library comes into play!

The eachSeries() function from Async.js is designed to execute asynchronous tasks one at a time. Here is how you can rewrite the iteration logic with using eachSeries():

const async = require("async");

// function asyncCall(timeToWait, dataToReturn) { ... }

console.log("START using Async.js");
async.eachSeries(
  values,
  async (value, callback) => {
    const data = await asyncCall(1000 * value, `data #${value}`);
    console.log(data);
    callback(); // signal completion of the current iteration
  },
  () => {
    console.log("END using Async.js");
  },
);

The new logic is much easier to read and maintain, eliminating the cumbersome Promise management introduced in the previous solution.

Once again, the output will be the expected one:

START using Async.js
data #1
data #2
data #3
data #5
data #8
END using Async.js

Wonderful! This is the power of Async.js.

Queuing Async Requests

To understand the capabilities of Async, let's consider another scenario.

Suppose you want to perform web scraping on a site with many pages. If you do not know what that entails, follow our guide on web scraping in Node.js. To speed up the process, the best practice is to send multiple simultaneous requests. At the same time, you do not want to overload the server with too many requests. So, you need to limit the number of concurrent requests in some way.

Now, imagine that you have a list of URLs that you want to scrape for data from:

const urls = [
  "https://example.com/page1",
  "https://example.com/page2",
  "https://example.com/page3",
  "https://example.com/page4",
  "https://example.com/page5",
  "https://example.com/page6",
  "https://example.com/page7",
  "https://example.com/page8",
  "https://example.com/page9",
  "https://example.com/page10",
];

To avoid flooding the site’s server, you want to have no more than 5 pending requests at the same time.

Implementing the desired behavior with vanilla JavaScript involves custom queuing logic. Instead, Async.js simplifies the process with its queue() method. This enables you to initiate new requests as soon as previous ones complete, as shown below:

const async = require("async");

// async function to scrape data from a URL
const scrapeUrl = async (url, callback) => {
  // scraping logic...
};

// list of URLs to scrape
const urls = [
  // omitted for brevity...
];

// create a queue with a concurrency limit of 5
const queue = async.queue(scrapeUrl, 5);

// add the URLs to the queue
urls.forEach((url) => {
  queue.push(url, (error) => {
    if (err) {
      console.error(`Failed to process ${url}:`, error);
    }
  });
});

// launch the scraping logic
queue.drain(() => {
  console.log("All URLs have been scraped successfully!");
});

When using queue() in Async, you first need to register all async tasks with the push() method. Then, you can execute them all simultaneously with the specified concurrency limit with drain(). Thanks to how a queue works, if the first request completes before the other four are still pending, the scraper will immediately send another request.

With Async, handling async queues has never been easier!

Conclusion

Async is a popular JavaScript library that addresses the limitations of native JavaScript's asynchronous programming features. For instance, it provides specialized versions of functions like map(), each(), every(), and reduce() that work well with asynchronous functions.

Here, you saw why async programming in JavaScript is not always a piece of cake. While you could address most issues with custom code, using a dedicated library like Async makes everything easier.

As shown above, integrating Async functions into your code is simple, whether you prefer the callback style or Promise-based syntax!

How to Map a JSON Column in Spring Boot With JPA 3 and Hibernate 6

Antonello Zanini — Thu, 24 Apr 2025 07:31:54 +0000

Building a RESTful application in Spring Boot that reads data from a database requires the definition of some ORM mappings. Even though this is generally a boring task, it is crucial for the proper functioning of the web application.

The ORM used by Spring Boot 3 with JPA 3 is Hibernate 6, which changed the way some column types need to be mapped. This is the case of JSON data type. In detail, you can now map a JSON column in Hibernate to a String, a Map, or a custom object.

Let’s learn the different approaches you can follow for JSON mapping in Hibernate 6 and JPA 3!

Getting Started

This article refers to Spring Boot 3 with JPA 3.1 and Hibernate 6.2. If you have a Spring Boot 2 project, follow the official guide to migrate it to version 3.0. Keep in mind that Spring Boot JPA 3 uses Hibernate 6 by default.

After setting up a Spring Boot 3 project, you will need a database involving JSON data to connect to. Any DBMS technologies supporting a data type for JSON will do. For example, suppose you have a configs table in a PostgreSQL database defined by the following two columns:

Note the data columns of type jsonb. That is the most complete data type for storing JSON data in PostgreSQL. Check out our guide to learn more about JSON vs JSONB in Postgres.

In detail, take a look at the query below:

This is the JSON data stored in a record of the config table:

{
  "graphics": {
    "resolution": "1920x1080",
    "quality": "high",
    "anti_aliasing": true
  },
  "gameplay": {
    "difficulty": "normal",
    "achievements": ["tutorial_passed", "chapter_1_mastered"]
  }
}

You now have everything you need to get started. There are three different approaches to JSON mapping in Hibernate. Time to explore them all!

Method #1: Map a JSON Column to a String

The easiest way to map a JSON column in Hibernate is to use a String. This is a straightforward and simple approach, but you should only use it when dealing with read-only JSON data. The reason is that changing and exploring the JSON data read at the application layer will involve cumbersome operations on a raw string.

You can map a JSON column to a String in a JPA entity as follows:

package com.dbvis.demo.entities;

import jakarta.persistence.*;
import org.hibernate.annotations.JdbcTypeCode;
import org.hibernate.type.SqlTypes;

@Entity(name = "configs")
public class Config {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    @Column(name = "id")
    private Long id;

    @JdbcTypeCode(SqlTypes.JSON)
    @Column(name = "data")
    private String data;

    // getters and setters omitted for brevity...
}

As you can see, to activate JSON mapping in Hibernate you only need to annotate your column attribute with @JdbcTypeCode(SqlTypes.JSON). Hibernate will now use Jackson, the standard JSON library coming with Spring Boot, to serialize and deserialize the attribute’s value to the type specified. In particular, it will convert data to a String when reading and transform it to JSON when writing.

Note that SqlTypes.JSON represent the generic SQL type JSON. Specifically, the default JSON data type used behind the scene changes according to the RDBMS dialect configured. For PostgreSQL, that is JSONB data type.

Now, assume you have a /api/configs/1 endpoint that returns the first record of the config table. Call it and you would get:

{
  "id": 1,
  "data": "{\"gameplay\": {\"difficulty\": \"normal\", \"achievements\": [\"tutorial_passed\", \"chapter_1_mastered\"]}, \"graphics\": {\"quality\": \"high\", \"resolution\": \"1920x1080\", \"anti_aliasing\": true}}"
}

As expected, the data field in the JSON response produced by Spring Boot is a string containing the JSON content.

Pros

Easy to implement
JSON schema agnostic

Cons

Basically limited to read-only JSON columns
JSON content is returned as a raw string

Method #2: Map a JSON Column to a Java Map

Another possible solution supported by Hibernate is to map the JSON content to a Map<String, String> object. Take a look at the JPA entity below:

package com.dbvis.demo.entities;

import jakarta.persistence.*;
import org.hibernate.annotations.JdbcTypeCode;
import org.hibernate.type.SqlTypes;
import java.util.Map;

@Entity(name = "configs")
public class Config {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    @Column(name = "id")
    private Long id;

    @JdbcTypeCode(SqlTypes.JSON)
    @Column(name = "data")
    private Map<String, String> data;

    // getters and setters omitted for brevity...
}

Notice the type declared on the data field. When retrieving JSON data, Spring Boot will now instantiate a Map object containing the JSON values in <key, value> string pairs. Similarly, Jackson will serialize the Map object associated with the data field to JSON when writing a new record to the database.

If you now call the /api/configs/1 API, you will get the following error:

com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)

That occurs because this approach to JSON mapping in JPA does not support JSON values involving arrays or nested objects. Thus, it only works with flat JSON data consisting of primitive fields.

Assume your data column stores the following flat JSON data:

{
    "difficulty": "normal",
    "quality": "high",
    "resolution": "1920x1080",
    "anti_aliasing": true
}

In this case, the endpoint will return:

{
  "id": 1,
  "data": {
    "quality": "high",
    "difficulty": "normal",
    "resolution": "1920x1080",
    "anti_aliasing": "true"
  }
}

Note that the data column containing JSON data was correctly serialized to JSON by Jackson. With this approach, the data field in the JSON response generated by the server is a proper object and no longer a simple string.

Pros

JSON content gets serialized as expected
Support for CRUD operation on the JSON content via the Map API

Cons

Only supports flat JSON data involving primitive values

Method #3: Map a JSON Column to an Object

To map a JSON column to a custom Java object, you only have to specify it as the field type:

import jakarta.persistence.*;
import org.hibernate.annotations.JdbcTypeCode;
import org.hibernate.type.SqlTypes;
// import the custom data type to map the JSON field to
import com.dbvis.demo.data.ConfigData;

@Entity(name = "configs")
public class Config {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    @Column(name = "id")
    private Long id;

    @JdbcTypeCode(SqlTypes.JSON)
    @Column(name = "data")
    private ConfigData data;

    // getters and setters omitted for brevity...
}

When querying some data, Jackson will now create a ConfigData object mapping the JSON content retrieved from the database. If you instead want to write JSON data, you will have to instantiate a new ConfigData object with the desired info. Jackson will automatically convert it to JSON for you.

ConfigData is nothing more than a POJO for data mapping. Defining POJO classes is a tedious process that involves boilerplate code. For this reason, you should use one of the many JSON to POJO converters available online.

In detail, this is what ConfigData looks like:

package com.dbvis.demo.data;

public class ConfigData {
    private ConfigDataGameplay gameplay;
    private ConfigDataGraphics graphics;

    // getters and setters...
}

Since the JSON data stored in the database contains two nested objects, you need two custom Java classes:

ConfigDataGameplay:

package com.dbvis.demo.data;

import java.util.List;

public class ConfigDataGameplay {
    private String difficulty;
    private List<String> achievements;

    // getters and setters...
}

ConfigDataGraphics:

package com.dbvis.demo.data;

import com.fasterxml.jackson.annotation.JsonProperty;

public class ConfigDataGraphics {
    private String quality;
    private String resolution;
    @JsonProperty("anti_aliasing")
    private boolean antiAliasing;

    // getters and setters...
}

Note the use of @JsonProperty Jackson annotation to specify custom mapping behavior. Do not forget that Hibernate will use Jackson for serialization and deserialization.

The /api/configs/1 endpoint will now return:

{
  "id": 1,
  "data": {
    "gameplay": {
      "difficulty": "normal",
      "achievements": [
        "tutorial_passed",
        "chapter_1_mastered"
      ]
    },
    "graphics": {
      "quality": "high",
      "resolution": "1920x1080",
      "anti_aliasing": true
    }
  }
}

Et voilà! This is exactly the JSON data contained in the database.

Pros

Support for any type of JSON data
Flexible and customizable
Enables typed JSON data management at the application level

Cons

Defining mapping POJO classes is cumbersome
Instantiating POJOs has an impact on performance
JSON data in the database must follow the schema defined by the POJOs

Explore JSON Data With DbVisualizer

As seen above, dealing with JSON with Hibernate is not that complex. At the same time, you may need a tool to explore the JSON data stored in your database. This is where an advanced database client such as DbVisualizer comes into play!

DbVisualizer is a powerful database management tool that supports a wide range of databases, such as MySQL, Oracle, Microsoft SQL server, PostgreSQL, and more. It comes with advanced data exploration features that allow you to visually see, edit, create, and delete data. Also, DbVisualizer has in-depth support for SQL data types, including JSON types.

Here are the steps to explore JSON data using DbVisualizer:

Connect to the database: Follow the wizard to set up a connection to your database.
Navigate to the desired table: Once you have connected to the database, use the “Databases” dropdown on the left to navigate to the table that contains the JSON data.

View the JSON data: Once you have navigated to the table, open it in a new tab, reach the “Data” tab, and double-click on the JSON data contained in some row. DbVisualizer allows you to view and edit it as a formatted JSON.

With DbVisualizer, dealing with JSON data has never been easier!

Conclusion

In this tutorial, you learned how to map JSON columns in Spring Boot. In detail, you saw three different approaches to represent JSON in Hibernate 6 and JPA 3. You can now map a JSON column to a String, a Map, and a custom Java object.

Also, you learned how important it is to have a tool to explore JSON data visually. Luckily, there is DbVisualizer, a fully-featured database client that comes with advanced data exploration capabilities. Try DbVisualizer for free!

FAQ

Let’s now answer some common questions about JSON mapping in Hibernate.

Is JSON column mapping possible in Spring Boot?

Yes, thanks to JPA and Hibernate, you can map JSON columns in Spring Boot. In particular, JPA 3 and Hibernate 6 have made JSON mapping easier and more straightforward.

Is it possible to define custom mapping behavior for JSON columns in Hibernate?

Yes, you can specify custom mapping behavior thanks to the annotations offered by Jackson or the JSON library configured in Hibernate by overriding the hibernate.type.json_format_mapper setting.

Is it possible to map JSON columns as non-string in Hibernate?

Yes, you can map JSON columns in Hibernate to different types. All you have to is specify the desired type on the @Column field. In detail, Hibernate supports Map and custom POJO type for JSON mapping.

Can I map nested JSON fields using JPA and Hibernate in Spring Boot?

You can map nested JSON fields in JPA and Hibernate by associating your JSON field in the JPA entity to a custom POJO class. This class can then involve fields of custom types. Basically, you have to break down your nested JSON structure into several data-mapping POJOs.

What are the benefits of mapping JSON columns in JPA?

Mapping JSON columns in JPA allows you to perform CRUD operations on those fields at the application layer. Also, if mapping the JSON columns correctly, JPA will automatically serialize and deserialize it for you.

How To Set Up Multiple Datasources in Spring Boot 3

Antonello Zanini — Thu, 24 Apr 2025 07:15:40 +0000

Separating data into different databases is a typical approach in the microservices world. This means that each backend usually needs to access only its limited database. At the same time, a backend application may need to use more than one database. This is why you can set up multiple datasources in Spring Boot.

In this step-by-step tutorial, you will learn how to configure Spring Boot Data 3 to work with many databases.

Getting Started

Suppose your Spring Boot RESTful backend needs to connect to the following two databases:

account: A MySQL database containing user data
company: A PostgreSQL database storing company data

The databases rely on two different RDBMS technologies and might even be on two different servers.

For the sake of simplicity, suppose that both databases has a single table. account has the user table and company has the product table. The goal here is to create a Spring Boot app that can use both databases. Let’s learn how to do it!

Keep in mind that this is just a sample scenario and the proposed solution can easily be extended to an indefinite number of databases, regardless of their DBMS technology.

Configuring Multiple Datasources in JPA 3

Follow this step-by-step tutorial and learn how to configure a Spring Boot 3 project to work with different databases in JPA.

Prerequisites

Before getting started, you need to add the following dependencies to your Spring Boot 3 project:

If you are a Maven user, add the lines below to your pom.xml file:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
    <groupId>com.mysql</groupId>
    <artifactId>mysql-connector-j</artifactId>
</dependency>
<dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
</dependency>

Otherwise, if you are Gradle user, verify that build.gradle contains:

implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
implementation 'com.mysql:mysql-connector-j'
implementation 'org.postgresql:postgresql'

The last two dependencies are the PostgreSQL and MySQL drivers, respectively. These allow you to connect to MySQL and PostgreSQL servers. Make sure to add the right drivers to your project’s dependencies according to the database technologies your backend relies on.

You now have everything you need to set up multiple datasources in Spring Boot. Time to learn how to do it!

1. Mapping Your Entities

Create an entities package and split it into the following two subpackages:

account: Will contain all MySQL database entities
company: Will contain all PostgreSQL database entities

Define the User and Product JPA entities and place them in the right subpackage.

2. Defining the Datasource Connection Properties

Add the connection info for the two target datasources to your application.properties file:

spring.datasource.account.url=jdbc:mysql://<MYSQL_DB_URL>/account
spring.datasource.account.username=<MYSQL_USERNAME>
spring.datasource.account.password=<MYSQL_PASSWORD>
spring.datasource.account.driver-class-name=com.mysql.cj.jdbc.Driver

spring.datasource.company.url=jdbc:postgresql://<POSTGRESQL_DB_CONNECTION_URL>/company
spring.datasource.company.username=<POSTGRESQL_USERNAME>
spring.datasource.company.password=<POSTGRESQL_PASSWORD>
spring.datasource.company.driver-class-name=org.postgresql.Driver

Replace all <___> parameters with the right strings. Note that account and company after spring.datasource are two custom options with the same name as the database they refer to.

3. Defining the Datasource Configuration Files

The info defined above can be used to create a database connection through a custom Spring Boot configuration file.

This is what the @Configuration file for the account database looks like:

package com.dbvis.demo.configs;

import com.dbvis.demo.entities.account.User;
import com.zaxxer.hikari.HikariDataSource;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.annotation.EnableTransactionManagement;
import javax.sql.DataSource;
import java.util.Objects;

@Configuration
@EnableJpaRepositories(
        basePackages = "com.dbvis.demo.repositories.account",
        entityManagerFactoryRef = "accountEntityManagerFactory",
        transactionManagerRef = "accountTransactionManager"
)
@EnableTransactionManagement
public class AccountDataSourceConfig {

    /**
     *  Initialize a DataSourceProperties bean by reading the
     *  datasource info from "spring.datasource.account.*" configs
     *  in application.properties
     */
    @Primary
    @Bean
    @ConfigurationProperties("spring.datasource.account")
    public DataSourceProperties accountDataSourceProperties() {
        return new DataSourceProperties();
    }

    /**
     * Create the datasource for the "account" database
     * using accountDataSourceProperties
     */
    @Primary
    @Bean
    @ConfigurationProperties("spring.datasource.account.configuration")
    public DataSource accountDataSource() {
        return accountDataSourceProperties()
                .initializeDataSourceBuilder()
                .type(HikariDataSource.class)
                .build();
    }

    /**
     * Configure Hibernate's EntityManager object
     * to look for Entity classes inside the "com.dbvis.entities.account" package
     */
    @Primary
    @Bean(name = "accountEntityManagerFactory")
    public LocalContainerEntityManagerFactoryBean accountEntityManagerFactory(
            EntityManagerFactoryBuilder builder,
            DataSource dataSource
    ) {
        return builder
                .dataSource(accountDataSource())
                .packages(User.class)
                .build();
    }

    /**
     * Create a transaction manager for the EntityManager object created with
     * the "accountEntityManagerFactory" bean
     */
    @Primary
    @Bean
    public PlatformTransactionManager accountTransactionManager(
            @Qualifier("accountEntityManagerFactory")
            LocalContainerEntityManagerFactoryBean accountEntityManagerFactory
    ) {
        return new JpaTransactionManager(
                Objects.requireNonNull(
                        accountEntityManagerFactory.getObject()
                )
        );
    }

}

This file reads the account connection info from application.properties to instantiate a DataSource bean. Then, it uses to create a valid EntityManager and TransactionManager. These two objects are what you need to handle a connection to a database. Check out the official documentation to learn more about how data access works in Spring Boot.

Note the use of the @Bean name property and @Qualifier annotation to tell Spring Boot what specific bean to use. Also, notice that the scope of the EntityManager gets restricted to the package containing the User entity, which is the account subpackage.

Similarly, you need to define a @Configuration file for company:

package com.dbvis.demo.configs;


import com.dbvis.demo.entities.company.Product;
import com.zaxxer.hikari.HikariDataSource;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.transaction.PlatformTransactionManager;
import javax.sql.DataSource;
import java.util.Objects;

@Configuration
@EnableJpaRepositories(
        basePackages = "com.dbvis.demo.repositories.company",
        entityManagerFactoryRef = "companyEntityManagerFactory",
        transactionManagerRef = "companyTransactionManager"
)
public class CompanyDataSourceConfig {

    /**
     *  Initialize a DataSourceProperties bean by reading the
     *  datasource info from "spring.datasource.company.*" configs
     *  in application.properties
     */
    @Bean
    @ConfigurationProperties("spring.datasource.company")
    public DataSourceProperties companyDataSourceProperties() {
        return new DataSourceProperties();
    }

    /**
     * Create the datasource for the "company" database
     * using companyDataSourceProperties
     */
    @Bean
    @ConfigurationProperties("spring.datasource.company.configuration")
    public DataSource companyDataSource() {
        return companyDataSourceProperties()
                .initializeDataSourceBuilder()
                .type(HikariDataSource.class)
                .build();
    }

    /**
     * Configure Hibernate's EntityManager object
     * to look for Entity classes inside the "com.dbvis.entities.company" package
     */
    @Bean(name = "companyEntityManagerFactory")
    public LocalContainerEntityManagerFactoryBean companyEntityManagerFactory(
            EntityManagerFactoryBuilder builder
    ) {
        return builder
                .dataSource(companyDataSource())
                .packages(Product.class)
                .build();
    }

    /**
     * Create a transaction manager for the EntityManager object created with
     * the "companyEntityManagerFactory" bean
     */
    @Bean
    public PlatformTransactionManager companyTransactionManager(
            @Qualifier("companyEntityManagerFactory")
            LocalContainerEntityManagerFactoryBean companyEntityManagerFactory
    ) {
        return new JpaTransactionManager(
                Objects.requireNonNull(
                        companyEntityManagerFactory.getObject()
                )
        );
    }

}

The main difference is that this config file refers to the company values in application.properties. Also, its Spring Boot beans are not marked with @Primary. This is because only one DataSource connection must be specified as primary.

If you forget to mark one of the two connections as primary, Spring Boot will fail with the following error:

Parameter 0 of method companyEntityManagerFactory in com.dbvis.demo.configs.AAProductDataSourceConfig required a bean of type 'org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder' that could not be found.

4. Creating the JPA Repositories

It only remains to define some Spring Data repositories. This will help you perform CRUD operations on your tables.

Create a repositories packages. Then, consider splitting the package into as many subpackage as the number of databases you need to connect to.

You can define a UserRepository as below:

package com.dbvis.demo.repositories.account;

import com.dbvis.demo.entities.account.User;
import org.springframework.data.jpa.repository.JpaRepository;

public interface UserRepository extends JpaRepository<User, Long> {}

Note that the JpaRepository does not require any extra configuration. This is because the AccountDataSourceConfig define earlier takes care of setting up everything required for the repository to work.

Similarly, define ProductRepository:

package com.dbvis.demo.repositories.company;

import com.dbvis.demo.entities.company.Product;
import org.springframework.data.jpa.repository.JpaRepository;

public interface ProductRepository extends JpaRepository<Product, Long> {}

You can now use the repository to perform operations on the account and company databases.

5. Putting It All Together

This is the file structure your project should now have:

ProductController uses the ProductRepository autowired object to retrieve data from the company database, while the UserController relies on a UserRepository instance.

Et voilà! Your Spring Boot 3 backend can now retrieve or write data to both the account and company databases, respectively.

Exploring Multiple Datasources in a Single Tool With DbVisualizer

If your project grows in complexity and requires multiple database connections, you will need an advanced tool to manage many datasources. This is what DbVisualizer is all about!

DbVisualizer is a complete database client that supports more than 50 different database technologies. This powerful tool enables you to take your database management to the next level. In particular, it comes with advanced data exploration capabilities, in-record data editing support, query optimization functionality, JSON and BLOB data type built-in support, and much more.

What are you waiting for? Try DbVisualizer for free today!

Conclusion

In this tutorial, you learn how to configure Spring Boot 3 to work with several databases. In detail, you saw how to use Spring Boot Data 3 to connect to multiple datasources. As shown here, it is not difficult and only requires some special configuration.

Knowing how to connect to multiple databases in Spring Boot is useful, but without the right data exploration tool everything becomes more complex. DbVisualizer is a MySQL, PostgreSQL, Oracle, and Microsoft SQL Server client that support more than other 50 database technologies. Thanks to its advanced features, managing numerous database connections simultaneously has never been easier. Download DbVisualizer for free!

FAQ

Let’s answer some popular questions on multiple database connections with Spring Boot.

How many databases can you connect to in Spring Boot?

You can connect to as many databases as the backend server can handle. There is no real limit to the number of database connections you can create with Spring Boot. At the same time, keep in mind that each connection to a data source takes resources. So, you should try to limit them to the bare minimum possible.

Can the same Spring Boot app use different database technologies?

Yes, Spring Boot Data supports the connections to several DBMS technologies, including both SQL and NoSQL technologies. You can connect to any database, as long as there is a supported Java JDBC driver.

How many database technologies can you use in a Spring Boot app?

Spring Boot Data does not have limits on the number of database technologies you can connect to. You can have a single DBMS, several connections to it, or a heterogeneous scenario with connections to different database technologies.

Can you simultaneously connect to both SQL and NoSQL databases with Spring Boot Data?

Yes, Spring Boot Data supports both SQL and NoSQL connections. Thus, you can set up both a MySQL and a MongoDB connection. For example, you can configure the Spring Boot application to work with SQL and MongoDB repositories.

Can two datasources have the same name in a Spring Boot app?

Yes, two datasources can have the same name. As long as they are on two different database servers and are configured properly at the application layer, Spring Boot can work with two or more datasources with the same name.

Best Testing Practices in Node.js

Antonello Zanini — Wed, 30 Oct 2024 12:54:41 +0000

Testing is a critical aspect of software development, as it ensures your application works as intended and meets quality standards.
In Node.js, testing is essential for the early detection of bugs in public endpoints.

However, there are many challenges associated with testing in Node.
External dependencies, asynchronous operations, and several possible input scenarios make writing tests a daunting task. Additionally, determining which components and aspects of your application to prioritize for testing can be challenging.

With some guidelines and best practices, you can tackle testing challenges and ensure that your backend is robust and reliable.

In this tutorial, we'll run through 15 top testing practices to write efficient, effective, and easy-to-maintain tests in Node.

Time to become a master of Node testing!

The Golden Rule of Testing in Node.js

Before diving into Node.js testing best practices, remember this golden rule:

Testing code is not production code.

Keep testing simple and avoid over-engineering. Tests should be intuitive and validate functionality, rather than implement complex design patterns or architectures.

Top 15 Node.js Testing Best Practices

Now that you've learned the golden rule of testing in Node.js, we'll explore 15 top testing best practices. Keep in mind that these tips apply to any Node test runner or testing library.

Let's dive in!

1. Consider the Test Pyramid

The Test Pyramid is a concept devised by Mike Cohn that first appeared in his book Succeeding with Agile, published in 2009.
This pyramid is a visual metaphor that explains how to approach testing.

Source: The Practical Test Pyramid

The Test Pyramid contains three layers. These represent what a test suite should consist of and the amount of testing to be performed on each level. The pyramid also highlights the expected performance and level of isolation required by each layer.

From a modern perspective, though, the test pyramid may appear overly simplistic. Even so, there are two lessons we can learn from it when designing Node.js tests:

Write tests with different levels of granularity.
The more high-level you get, the fewer tests you should have.

2. Define Easy-to-Understand Test Titles

Engineers or DevOps experts who take care of deployments may not be familiar with code. When a deployment fails because of a failed test, their first reaction will be to look at the logs or test reports. In this case, the only thing they might see is the name of the failed test.

That's why each test should have a title that is informative enough to explain exactly what the test does. Specifically, a test name should answer these three questions:

What is being tested?
What is the expected result?
Under what circumstances? / In what scenario?

As a reference, take a look at the Node.js test below:

// unit/component under test (1.)
describe("user authentication service", function () {
  describe("login functionality", function () {
    // expected result (2.) and circumstance/scenario (3.)
    it("should reject login with incorrect password", () => {
      // ...
    });
  });
});

3. Include Tags with Your Test Titles

Adding tags to test titles improves test organization. Tags such as "#api" or "#authentication" help categorize tests and allow you to run similar tests in batches, because Node test runners can usually execute tests that contain a specific string in their titles. Also, by using tags, it's easier to find tests using the search capabilities of your IDE.

4. Follow the Arrange, Act, Assert (AAA) Pattern in Your Tests

The Arrange, Act, Assert pattern ensures that each test has a clear scope and is well-structured. A test written according to this pattern should be based on these three phases:

Arrange: Set up the test environment. This also includes adding records to the test database and defining any necessary stubs and/or mocks.
Act: Execute the code that will be tested. That generally boils down to calling a single function or method.
Assert: Verifies the expected outcomes.

This approach to test writing enhances readability and maintainability.

5. Focus on Testing Public Behavior

A well-known and effective strategy for API testing is to focus on public behavior. At the end of the day, the main goal of your testing operation is to make sure that your Node endpoints produce the expected results when given specific inputs. This contract-based approach to testing, also known as black-box testing, aims to test the expected input-output relationships without delving into internal implementation details.

Libraries like node-mocks-http enable the creation of req and res objects to call routing functions and simulate incoming requests.
You can then inspect the resulting res object to check API behavior, such as via JSON schema validation. Additionally, you can assert whether the HTTP status or headers set on the res object align with expectations.

By targeting public behavior, you can verify that your APIs adhere to defined contracts with just a few tests. This approach checks that your Node.js endpoints behave as intended for end users, without you having to test dozens of functions.

6. Use a Dedicated Database in Each Test

Using dedicated databases in every test is key to ensuring isolation and reproducibility. The biggest objection you might have is that this approach leads to data duplication. Well, don't forget the golden rule!

Employing shared databases for multiple tests can be the source of flaky behavior. A test might fail because the record it's supposed to operate on has been modified or deleted by another test running at the same time.

Setting up separate databases for each test prevents data contamination and interference between tests. This leads to robust test results and opens the door to safe test parallel execution.

7. Define an Effective Data Clean Strategy

The AAA pattern and previous best practices recommend that each test sets up its own database. To prevent filling your memory or disk with unnecessary data, you must define an effective data-cleaning strategy.

There are a few possible options for data cleaning, including cleaning the database:

After every single test.
After all tests have finished.
Periodically, at set intervals.

The right strategy depends on your specific project requirements and architecture. Regardless of the approach selected, don't clean the database on test failures. Having access to the data associated with a failed test is essential for debugging and understanding what went wrong.

8. Use Realistic Input Data

When testing methods and functions that need input data, you might be tempted to use random strings and values like "foo" or "1234."
While this reduces the mental effort of coming up with accurate input, it's a bad practice. Instead, always test functions with realistic inputs. This way, you can ensure that your Node.js tests are more representative of real-world scenarios.

For automatic data generation, consider using libraries like Chance or Faker.
These can generate fake — yet realistic — data resembling production data, such as real-world phone numbers, usernames, credit card numbers, company names, and text.

9. Use Property-Based Testing to Cover All Possible Input Combinations

While you can test a function with realistic data, testing all possible user input combinations may take too much time, especially when accepting many inputs. The problem is that one of those untested combinations might lead to an unexpected result. That's where property-based testing comes in!

This technique involves automatically testing a function with a wide range of input combinations, increasing the chances of discovering bugs. Instead of manually selecting a few input samples, property-based testing generates numerous permutations to thoroughly examine how your code handles different scenarios. This helps identify edge cases and unexpected input interactions, detecting issues that might not emerge with manual testing.

For instance, the function addNewProduct(id, name, isDiscount) can be tested with multiple combinations of (number, string, boolean) such as (1, "iPear", false), (2, "Universe XL", true), or (0, undefined, null). By using libraries like fast-check, you can automate property-based testing within your favorite test runner.

10. Prefer Stubs and Spies Over Mocks

When using test doubles, prefer stubs and spies over mocks to maintain simplicity and clarity. Stubs replace functions and control their behavior, while spies track function calls and arguments. Both are less intrusive and easier to manage compared to mocks, which often involve complex logic to interact with the other components in your application.

Libraries like Sinon.JS provide robust support for stubs and spies, enabling precise control and inspection of your code's interactions. By favoring stubs and spies, you can keep your tests concise, easier to understand, and less prone to errors.

11. Avoid Catching Expected Errors

Some tests verify whether a function throws a specific error on faulty input. A common mistake when writing such tests is surrounding the function call with a try ... catch statement and asserting whether the test entered the catch branch.

Avoid that, as it only makes the test logic verbose and more difficult to read and maintain. Instead, prefer dedicated assertions like expect(<function call>).to.throw(<blank or error type>) in Chai or expect(<function call>).toThrow(<blank or error type>) in Jest. These assertions provide a clearer and more elegant way to test for thrown errors without catching them.

12. Don't Forget to Test Middleware Functions

As mentioned in testing tip 5 above, you should focus on testing public behavior. However, you shouldn't ignore internal code entirely. For instance, dozens of endpoints exposed by your backend may rely on the same middleware. A bug in that middleware would introduce faulty behavior across many APIs!

While tests for endpoints will fail because of that bug, understanding that the cause of all those failures is down to the same middleware may not be easy. Given their vital importance in a Node.js architecture, it makes sense to have tests dedicated to middleware functions.

A middleware can be tested like any other function. Forge req and res objects with libraries like node-mocks-http and verify that it performs the required actions on these objects. This way, you can guarantee that your middleware layer is reliable and doesn't introduce errors to your endpoints.

13. Produce Useful Test Coverage Reports

Most Node.js testing technologies come with reporting capabilities.
This means they can be configured to generate reports in HTML, XML, or other formats after a test suite run. These reports highlight test results and provide useful statistics.

The reports prove invaluable for QA engineers, who may not be familiar with a codebase, by identifying failing tests and areas of an application that require further tests. A key metric in these reports is code or test coverage, which summarizes the percentage of your codebase exercised during testing.

Coverage reports provide insights into the quality and completeness of your tests, highlighting areas and components that require additional attention. By analyzing and monitoring the results of these reports, you can identify untested code paths and monitor the evolution of your testing operation. You should always configure your testing library to produce test coverage reports.

14. Run Performance and Stress Tests

Writing tests in a Node.js application isn't always about test coverage. You also need to guarantee that your backend remains responsive and fast, even under stressful conditions. You can achieve that by running performance and stress tests. That ensures that your application can handle an unexpected workload and maintain optimal performance under various traffic conditions.

Performance tests evaluate response time, throughput, and resource usage to help you identify bottlenecks. Stress tests simulate heavy loads and peak traffic scenarios to assess system stability and resilience. By regularly performing these types of tests, you can proactively identify and address performance issues and prevent downtime.

15. Refer to Other Popular Best Practices

Staying up-to-date on popular best practices is critical.
Refer regularly to resources such as the Node.js docs, trusted blogs, and community forums to make sure that your tests remain robust, secure, and efficient. Don't forget that this, or any other, guide is only an introduction to the complex world of Node.js testing best practices.

Wrapping Up: Improve Your Node.js Testing Process

In this blog post, we took a look at the golden rule of Node.js testing and explored some of the most relevant best testing practices.

You now know:

To consider testing code not as production code
What components in your Node.js application you should focus your testing efforts on
Many tips and tricks to keep your tests simple, effective, and easy to maintain

Thanks for reading!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

DbVisualizer 24.2: A Complete Review

Antonello Zanini — Mon, 23 Sep 2024 14:01:52 +0000

Discover what DbVisualizer 24.2 has to offer in this in-depth review and learn why it represents the biggest DbVisualizer release to date

DbVisualizer 24.2 marks a major step for the database client with the highest user satisfaction rate on the market, taking it to the next level thanks to a refined user interface and many other improvements.

In this review, you’ll discover why DbVisualizer underwent a UI redesign, what new features and improvements version 24.2 has introduced, and how to try this latest release.

Let’s find out what an experienced developer and database administrator thinks of DbVisualizer 24.2 in my review!

The Need for a Restyling of DbVisualizer

As you may already know, DbVisualizer has been the database client of choice for years according to top companies, government agencies, and individual developers.

According to the official site, over the years, more than 6 million users have downloaded the software, and more than 28,000 companies spanning 145 countries are currently using the commercial Pro version.

The company proudly states that Tesla, NASA, Apple, Netflix, and many other leading companies have chosen DbVisualizer for their database operations.

No surprise, over 216 users on G2 and more than 96 users on Capterra have given it top-notch positive reviews. With a rating of 4.6/5 on G2 and 4.8/5 on Capterra, DbVisualizer has the highest user satisfaction rate in the market. Try DbVisualizer, and you’ll agree with them!

The secret of its success? Well, it must be the support for more than 50 databases and a Java core, which makes the tool multi-platform and extremely reliable.

However, in my opinion as an experienced developer, what sets DbVisualizer apart from other database clients is its extensive range of capabilities. The software offers a stunning number of utilities, options, and functionalities, appealing to both beginners and professional database administrators with years of experience. That’s impressive!

Actually, you can easily tell that the development team behind DbVisualizer has focused their efforts on providing robust, powerful, easy-to-use, and time-saving features.

With DbVisualizer 24.2, the team appears to have put the same dedication into improving the UI and UX, aiming to elevate the database client we all love to the next level.

Time to find out more about this new version of DbVisualizer!

DbVisualizer 24.2: The Biggest Update So Far

Released in June 2024, DbVisualizer 24.2 has been presented as the most significant update in the history of DbVisualizer (as of this writing, at least).

This new version introduces numerous changes, with the most noticeable being the revamped user interface:

As you can see, that’s definitely a remarkable improvement!

The new UI now appears more intuitive, modern, and user-friendly. In addition to the impressive new interface and updated themes, the 24.2 release introduces:

7 new features
17 improvements
11 bug fixes

These additions include support for persistent connections via the dbviscmd CLI tool, dedicated actions to select the previous/next statement in the SQL editor, support for Azure Synapse Analytics, enhanced autocomplete capabilities, full support for roles in PostgreSQL, automatic theme switching, support for Databricks, and many others. Those are a lot of new features!

Discover all the changes provided by DbVisualizer 24.2 in the release notes.

What’s New in DbVisualizer 24.2?

Easy question! Just have a look at the summary table below. You’ll discover the most important changes introduced in this major release of DbVisualizer:

Type	Title	Description
New Feature	UX/UI	Full window support on macOS. Automatic theme switching, new dark and light themes, new scalable icons
New Feature	Command Line Support (`dbviscmd`)	Add support for creating persistent database connections using `dbviscmd`
New Feature	DB Support: Azure Synapse	Add support for Azure Synapse Analytics
New Feature	DB Support: Databricks	Add basic support for Databricks
New Feature	DB Support: MariaDB/Oracle	Navigate between procedures and functions in the package body editor
New Feature	Export	Add support for generating `MERGE` statements when exporting data as SQL and for trimming text values when exporting data
New Feature	SQL Editor	Add actions to select the previous/next statement in the editor (`Ctrl+Alt+Up/Down`)
Improvement	Auto-Completion	Automatically qualify columns when the column name alone is ambiguous
Improvement	Cell Viewer/Editor/Grid Component	Add support for SVG images in the cell viewer and the data grid
Improvement	Create/Alter Table/DB Support: PostgreSQL	Add support for sequence names in auto-generated columns in PostgreSQL
Improvement	DB Support: PostgreSQL	Add support for roles in PostgreSQL (replacing users and groups)
Improvement	DB Support: Redshift	Add support for materialized views in Redshift
Improvement	General	Improved support for arrays
Improvement	Security	Add the option to reset the master password also when a master password is required
Bug Fix	Create/Alter Table/DB Support: PostgreSQL	`NOT NULL` can be omitted in the DDL for auto-generated columns in PostgreSQL
Bug Fix	DB Support: ClickHouse	Can't list databases in older versions of ClickHouse
Bug Fix	Export/Import User Settings	Passwords are not included when exporting/importing SSH configurations
Bug Fix	Query Builder	When loading a query containing an unsupported operator into the query builder, it’s replaced with "="

Let me now review each of the most interesting changes introduced by DbVisualizer 24.2!

A Fresh New UI

In release 24.2, DbVisualizer’s UI has been significantly refined. As the GIF below suggests, that’s not just a color adjustment but a complete UI overhaul.

According to the release notes, the goal of the development was to create a balanced work environment for both light and dark theme users.

Well, I think they achieved their goal brilliantly! The new UI just looks fresh and intuitive.

The dark theme has become darker, and the light theme has become lighter, with DbVisualizer that can now automatically match the dark or light setting of your OS.

You can tell that the team worked hard to balance the look and colors, as well as introduce small improvements to margins around elements and line heights in lists.

Another improvement made by the team in DbVisualizer 24.2 is a new set of icons. They replaced the old icons with modern and scalable SVG icons, ensuring they look good on any screen and resolution settings:

To avoid disappointing or disorienting DbVisualizer veterans, the organization of UI elements remains largely unchanged. Only the placement of some buttons has been improved to enhance usability and make daily use more intuitive. (Thank you for considering us, DbVisualizer team!)

Check out the docs to see how to get the most out of the new UI.

Improved Database Support

Below are the database support improvements added in version 24.2:

Azure Synapse Analytics: DbVisualizer now has extended support for dedicated and serverless SQL pools in Azure Synapse Analytics. That includes support for database-scoped credentials, external file formats and data sources, and external tables. For more information, see the Azure Synapse Dedicated and Azure Synapse Serverless pages on the official site.
Cassandra: Added support for the JDBC wrapper of the Java driver for Cassandra, which is more powerful and future-proof than previous alternatives. Moreover, other Cassandra-related improvements have been implemented, such as support for overloaded functions and error marking in the Procedure Editor.
Databricks: Introduced basic database support for Databricks, with a pre-defined entry for Databricks in the driver manager for simplified setup.
Greenplum: Database integration extended to Greenplum 7, including adjustments for partitioned tables, generated columns, constraints, procedures, functions, aggregates, and sequences.
MariaDB: New navigation panel added to the Package Body Editor, making it easier to navigate between procedures and functions in a package.
MySQL: Added the possibility to create and update generated columns in the Create Table and Alter Table dialogs.
Oracle: The same improvement added for MariaDB has also been introduced for Oracle.
PostgreSQL: Support for PostgreSQL has been extended to include role management. Also, users can now specify the sequence name when defining auto-generated columns.
Redshift: Added support for materialized views in Redshift.
Snowflake: Introduced support for additional table types in Snowflake, including dynamic tables, event tables, hybrid tables, and iceberg tables.

Enhanced Data Export Options

Among the tons of features supported by DbVisualizer is the ability to export databases/schemas and tables.

The first option is ideal for exporting DDL for objects such as tables, views, procedures, functions, triggers, packages, and package bodies in a database/schema. The second option is instead dedicated to exporting table data in various formats, including as an SQL file or to the system clipboard.

With DbVisualizer 24.2, when exporting data in SQL format, users can now select between INSERT statements and MERGE statements:

Compared to INSERT statements, MERGE statements allow for the merging of data from the source table to the target table.

That ensures that existing rows in the target table are updated, while new rows are inserted, offering an efficient and robust way to synchronize data between tables.

Support for text functions has also been added to the data export dialog. When selected, the text function will be applied to all text columns that are exported. Currently, the available options are:

Keep the text as is
Trim both ends
Trim the right end
Trim the left end

Another tweak introduced in the new version of DbVisualizer I found amazing is that, when exporting data to a file, the correct filename extension is now appended automatically depending on the selected output format.

When the user selects a format, the export file name is updated accordingly with the appropriate extension. That saves you a lot of unwanted errors, trust me!

As an additional improvement, when previewing the export in SQL format, syntax highlighting is also applied to the generated code to make it easier to read.

Refined SQL Editor

As all DbVisualier users know, one of its most beloved features is its fully-featured SQL editor.

In release 24.2, tthe editor has been further improved with several enhancements:

Advanced substitution in Find and Replace: The Find and Replace panel now includes buttons to add newlines and supports literals \n and \t. Also, it accepts \L and \U for lowercase and uppercase manipulation, respectively.
Smarter auto-completion: When auto-completing column names, these will now be automatically qualified if the column name alone is ambiguous. See the details in the documentation.
Select next/previous statement: Added support for the CTRL+ALT+UP and CTRL+ALT+DOWN shortcuts to select the next and previous statements in the current SQL script, respectively.
Support for a list of choices in custom variables: Introduced the possibility of defining variables with a list of available values for advanced scripting.

Other Changes Worth Mentioning

While I would love to dig into each of the updates and improvements added by DbVisualizer 24.2, they’re just too many. Thus, here are some other noteworthy changes:

Improved support for arrays: DbVisualizer now supports a JSON-based syntax for presenting and editing arrays. Additionally, it has extended support for understanding the content and structure of array objects. This enables you to validate the input before sending data to the database.
Uncommitted SQL statements: When warning about auto-commit being turned off, DbVisualizer now displays which statements will be committed as a result of the commit.
Special characters in text fields: Newlines and tabs in text fields are now rendered as symbols in the Data grid.
Images in the Data grid: Binary columns containing SVG images are now displayed in the Data grid. The team has also implemented multi-threading rendering for faster image loading.
Persistent database connections via CLI: The DbVisualizer CLI now supports the creation of persistent database connections via the command line.

To learn how to make the most of all new features, check out DbVisualizer 24.2 User Guide.

Is 24.2 the Best Version of DbVisualizer?

Yes, I have no doubt that version 24.2 is the best release of DbVisualizer ever!

That’s clear when you consider the pros and cons of this new version:

👍 Pros:

A fresh, modern UI
New SVG icons
Over 10 bug fixes for a more reliable experience
Support for Azure Synapse Analytics
Improved SQL editor
Enhanced autocomplete capabilities
Improved support for arrays
New data export options
Persistent connections via dbviscmd in the command line
Basic support for Databricks
Support for SVG images in the Data view
Improved Find and Replace feature
New shortcuts

👎 Cons:

Adapting to the new UI might take some time

The list of advantages outweighs the drawbacks. The only notable disadvantage is the adaptation to the new UI, which might throw into confusion some veteran users (like me). Still, that’s the inevitable price to pay for a revamped and more modern user interface.

At the end of the day, our habits as users change as technology evolves, and software UIs must evolve accordingly.

The good news is that the team has introduced other tweaks and UX optimizations to greatly reduce the feeling of disorientation. Surprisingly, the new UI just feels familiar. Give it a try, and you'll see what I mean!

How To Try DbVisualizer 24.2

If you’re already a DbVisualizer user, simply upgrade the software to the latest version. If you need assistance with this process, refer to the guide in the official documentation.

If you want to test DbVisualizer 24.2 as a new user, you can download it for free. That’ll give you access to essential features, documented in a helpful User Guide and an active Community forum.

To access all DbVisualizer features, consider that you need to upgrade to the Pro plan. Upon purchase, you’ll also receive additional guidance from the development team for the first 60 days.

For enterprises, businesses, and individuals needing priority assistance throughout the subscription period, there’s a special Pro plan available.

For more details, visit the pricing page on the official website or explore the comparison between the Free and Pro plans.

To test all the features of the Pro version, you can also evaluate DbVisualizer tool for 21 days via a free trial.

Final Verdict

In this review, I had the chance to try out DbVisualizer 24.2 and see why it’s a huge leap forward for one of the most loved database clients out there.

The development team didn’t just build on its already great features—they’ve also polished the user interface to meet modern standards of usability and style. This perfect mix of functionality and design keeps DbVisualizer at the top for both seasoned pros and newcomers in the database world.

If you’re excited to explore DbVisualizer 24.2 further, check out the links below:

I can confidently say that version 24.2 is the best DbVisualizer has ever been. Whether you’re upgrading from an older version or checking it out for the first time, now is a great time to experience its new features.

Review score: 9.5/10

How to Handle Errors in Next.js for Node With the App Router

Antonello Zanini — Wed, 11 Sep 2024 10:11:18 +0000

Error handling in Next.js is critical to providing a seamless experience to your users even when things go wrong. Without proper error management, users may get confused about what has happened and even leave your site. To avoid that, you must ensure that they receive informative feedback about errors and provide a way to recover from them.

In this article, you'll see:

Why the default Next.js error handling logic falls short
How the error.js file convention allows you to handle errors in the App Router
How to define custom logic to deal with both client-side and server-side errors in Next.js

Let's jump right in!

Default Error Handling in Next.js

When a server error occurs in a Next.js application, the following '500 Internal server error' page is returned by default:

Instead, this view is loaded when a fatal client-side error occurs:

There are two huge issues with this behavior:

Generic error messages: Both the "Internal server error" and "Application error: a client-side exception has occurred (see the browser console for more information)" messages are too generic for you to understand what happened.
No recovery: Users don't have the opportunity to recover from the error and are forced to manually reload the page or leave.

As a result, the default Next.js error handling system leads to a pretty bad user experience. Here's why you need to override that with custom error handling logic!

How to Handle Errors in Next.js Using the App Router

Error handling in a Next.js application with the App Router revolves around the error.js file convention.
If you aren't familiar with that, error.js is an optional file inside a route segment that exports a UI React component to handle errors gracefully.

To better understand how this mechanism works, let's analyze two possible scenarios:

Local and nested errors
Root-level errors

Local Error Handling

Assume that you defined an error.js file in a route segment:

When generating the /dashboard page, the App Router will automatically create a React Error Boundary and wrap the page.js component as follows:

Now, consider if an error occurs in page.js or any of its nested child route segments. That can be either a client-side or server-side error (as error.js can deal with any type of error).

The Error Boundary will intercept the error and render the React component exported by error.js as a fallback. In particular, the error component will be rendered within the surrounding layout. This means the error component won't take the entire view as it does in the default Next.js error handling logic, but occupy only a portion of the page. That leaves room for recovery, as the layout will maintain its state and remain interactive.

That's why the component exported by an error.js file accepts two props:

error: Contains the Error instance with the details of the error that occurred in the client or the server. In production, server-side errors include only generic message and digest properties to avoid leaking sensitive information.
reset: A function to attempt to recover from the error. When called, the function tries to re-render the original component nested in the Error Boundary. If successful, the fallback error component is replaced with the newly rendered result. Otherwise, the fallback error component is left on the page.

This error handling mechanism works on a per-route-segment basis, with errors bubbling up to the nearest error boundary. By placing error.js files at different levels in the nested hierarchy, you can then achieve more or less granular error UI management.

Root Error Handling

error.js boundaries can't catch errors occurring in layout.js or template.js components of the same segment. Instead, those errors will be intercepted by the error.js file in the parent segment.

But what happens if an error is thrown in the root app/layout.js or app/template.js component? Since the root app/error.js boundary won't be able to catch it, Next.js requires a special component. To specifically handle errors in the root layout.js and template.js components, you must place a global-error.js file in the root app directory. That is nothing more than a variation of error.js, exporting a component with the same props.

The main difference with error.js is that the global-error.js error boundary wraps the entire application. So its fallback component will replace the entire view and should always contain the <html> and <body> tags.

global-error.js is the least granular UI component and can be considered the last resort for error management. Although it's essential to ensure complete Next.js error handling, it's unlikely to get rendered often, as the root layout and template components are generally static and less prone to errors.

Implement Error Handling in Next.js

Follow the instructions below to handle errors in Next.js with error.js files.

Prerequisites

To go through this tutorial, you need Node.js 18.17 or later installed on your machine. In particular, the code snippets below will refer to a Next.js 13+ application with the App Router.

For access to the entire code of the project you're about to build, clone the GitHub repository that supports this article:

git clone https://github.com/Tonel/custom-error-handling-nextjs

To test the application, enter the project folder and install the local dependencies:

cd custom-error-handling-nextjs
npm install

Then, build the Next.js application:

npm run build

This will take a while, so be patient.

Once it finishes, you can launch the application with:

npm run start

The demo Next.js application with custom error handling logic should now be running at http://localhost:3000.

If you prefer to start from scratch, launch the create-next-app command and follow the instructions to set up a new Next.js application.
Refer to the GitHub repository for the complete code, as the sub-chapters below will cover only the main steps of implementing error handling in Next.js.

Create An `error.js` File

Add an error.js file to the root app directory and initialize it as below:

// ./src/app/error.js

"use client";

import styles from "./error.module.css";

export default function Error({ error, reset }) {
  // custom logic (e.g., log the error or send it to an APM service)

  return (
    <div className={styles.error}>
      <div className={styles.oops}>Oops!</div>
      <div className={styles.message}>Something went wrong...</div>
      <div>
        <button className={styles.retryButton} onClick={() => reset()}>
          🔄 Retry!
        </button>
      </div>
    </div>
  );
}

error.module.css is a local CSS module containing the styling for the fallback error component. Inside these error UI components, you can also add custom logic, such as logging and application monitoring integrations. In a real-world scenario, you should provide a meaningful explanation for the error.

Keep in mind that error components must be client components. If you forget to add the "use client" instruction, Next.js will raise this error:

ReactServerComponentsError:
./src/app/error.js must be a Client Component. Add the "use client" directive to the top of the file to resolve this issue.
Learn more: https://nextjs.org/docs/getting-started/react-essentials#client-components

This is what the <Error /> component returned by error.js looks like:

As you can see, it presents the error gracefully to the user and allows them to recover through the "Retry" button.

Create a `global-error.js` File

Create a global-error.js file in the app directory and add the following lines to it:

// ./src/app/global-error.js

"use client";

import Link from "next/link";

export default function GlobalError({ error, reset }) {
  // custom logic (e.g., log the error or send it to an APM service)

  return (
    <html>
      <body>
        <div className={"globalErrorContainer"}>
          <div className={"globalErrorDiv"}>
            <div className={"globalError500"}>
              <h1>500</h1>
              <h2>Internal Error</h2>
            </div>
            <Link className={"home"} href={"/"}>
              Home Page
            </Link>
          </div>
        </div>
      </body>
    </html>
  );
}

If the application crashes due to a fatal error in the root layout or template files, a reset() call is unlikely to resolve the situation.
As a recovery strategy, you should instead add a link to the home page or force a page reload with window.location.reload();.

Note that Next.js will ignore any CSS module or global file imports in global-error.js.
To properly style this component, add the required CSS classes to your global root CSS file.

Here's how the <GlobalError /> component appears:

In a real-world site, replace "Internal Error" with a better error message.

Bear in mind that Next.js ignores the global-error.js file in development mode.
In that scenario, a fatal error in the root layout or template component triggers an overview with detailed debugging information, like this:

To test the error component returned by global-error.js, you must first build your application and then run it with npm run start.

Get Ready to Trigger Errors

To verify that error.js and global-error.js work as expected, define a <ErrorButton /> component:

// ./src/app/components/ErrorButton.js

"use client";

import { useState } from "react";
import styles from "./error-button.module.css";

export default function ErrorButton(props) {
  const [raiseError, setRaiseError] = useState(false);

  if (raiseError) {
    // "a" is undefined so "props.a.b" will result in an error
    return props.a.b;
  } else {
    return (
      <button
        className={styles.errorButton}
        onClick={() => setRaiseError((error) => !error)}
      >
        {props.label}
      </button>
    );
  }
}

When clicked, raiseError becomes true and a client-side error is thrown.

Now, nest that component in layout.js to trigger global errors and in page.js to trigger local errors.
Here's what the home page will contain:

Wonderful! You're now ready to test the custom Next.js error handling logic!

Put It All Together

Launch the complete Next.js application as described in the Prerequisites section of this post.

Open http://localhost:3000 in the browser and click the "Simulate Local Error" button:

The <Error /> component will be rendered in the global layout UI as desired.
By clicking "Retry," the App Router will re-render the page.js component from scratch.
This means that the raiseError state property from <ErrorButton /> will be assigned to false and the error will be recovered.

Now, click the "Simulate Global Error" button:

This time, the <GlobalError /> component will replace the entire view as expected.

Et voilà! You are now a Next.js error handling master!

Error Handling in Next.js: Best Practices

Handling errors in Next.js can be challenging, especially if you don't follow these best practices:

Always define a global-error.js file: Even though the error.js boundaries will catch most errors, your project must have a global-error.js file to make sure that even fatal errors at the root layer are handled gracefully.
Don't use global-error.js as a replacement for error.js: Even if your project has global-error.js in place, you should define a root error.js to handle non-root errors within the global layout.
Add recovery logic: Your users should always have the opportunity to recover from the error situation.
Avoid leaking implementation details: The error message should be detailed enough to explain what happened to the user. At the same time, it shouldn't provide implementation or sensitive information to potential attackers.

Wrapping Up: Next.js Error Handling Made Easier!

In this blog post, you saw how to introduce custom error handling logic to your Next.js application using the App Router, including:

How Next.js handles errors by default
How to define error UI components in the App Router
The best practices for secure and effective Next.js error handling

Thanks for reading, and see you in the next one!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Expand Content Reach Using AI for SEO and Translation in Your CMS

Antonello Zanini — Thu, 22 Aug 2024 15:38:48 +0000

In recent months, the way we approach our work has changed significantly. AI has become central to many workflows, driving productivity, enhancing efficiency, and providing valuable insights. The CMS industry is no exception to this trend. That is why why Apostrophe recently released two new extensions that leverage AI for SEO and translation.

In this article, you will discover the latest developments from Apostrophe in the AI space. You will learn what the new SEO Assistant and Automatic Translation extensions are, and see how they can streamline your SEO metadata definition and content translation processes with just a few clicks.

AI Applications in the CMS Industry: From Ideas to Reality

In our article published in mid-2023, we examined how AI could transform the CMS industry. During that discussion, we introduced the AI Helper extension to generate images and rich text directly from a prompt within ApostropheCMS.

In the months that followed, those insights were validated and evolved into concrete developments. These ideas were incorporated into Apostrophe's roadmap and have now materialized into two new AI-powered extensions for Apostrophe Pro.

Let’s dive into these exciting new features!

Introducing New Apostrophe AI-Powered Plugins: SEO Assistant and Automatic Translation

These are two new extensions in the Apostrophe AI offerings:

SEO Assistant: Automatically generates SEO page metadata through AI.
Automatic Translation: Provides AI-driven translation of documents (pages and pieces) for content localization.

Note: Both extensions are exclusive to Apostrophe Pro and Apostrophe Assembly users. These premium versions of Apostrophe offers advanced features tailored for both developers and editors.

As you are about to learn, both extensions work together to optimize your site’s reach. To better showcase SEO Assistant and Automatic Translation, they will be integrated into the site showcased in our tutorial on how to create an e-commerce platform with Apostrophe. If you are not familiar with that article, you can see an example of in the following video.

Using AI for SEO Optimization in Your CMS

One of the most tedious, yet essential, tasks when handling the SEO of a website is defining the meta title and SEO description of a webpage. This information is key because search engines like Google use it to generate the SERP (Search Engine Results Page) entry associated with your page, as in the following image:

This is even more important when considering that over 50% of traffic to business websites comes from organic search (i.e., SEO).

When specifying the SEO meta title and description of a page, there are a few best practices to keep in mind:

Both the title and description should be concise, actionable, and to the point. They should also include relevant keywords to help search engines rank your page higher.
The meta title should be between 50 to 60 characters long to ensure that search engines do not truncate it in the SERP snippet.
The meta description should ideally be between 50 to 160 characters, with an optimal length of around 150/160 characters. For more information, check out Google's best practices of meta descriptions.

The problem is that writing effective SEO titles and descriptions is often seen as a boring task. Ask your editors, and they will likely tell you that they prefer to craft great content rather than focusing on SEO details.

Additionally, SEO optimization is typically the final step in the publication process. As a result, some editors might overlook it or not give it the attention it deserves to push the page online.

The good news is that SEO title and description values are directly related to the page's content, which opens the door to leveraging AI for SEO. Specifically, the AI-powered SEO Assistant extension can analyze the page content to generate effective meta titles and descriptions for you.

Let’s explore how this Apostrophe Pro extension works in the section below!

SEO Assistant in Action

Follow the official integration guide to install and configure the SEO Assistant extension in your Apostrophe application. As of this writing, the prerequisites for using the SEO Assistant are:

An Apostrophe 4 application with the SEO Tools and other required extensions installed, as described in the documentation.
An OpenAI API key or a custom integration.

In this example, we will use OpenAI as the AI provider, but keep in mind that SEO Assistant also supports custom AI providers.

Now, assume you just created a special web page called “Summer Collection 2024" to promote your clothing line for the summer season. The final step before publishing it is to define the meta title and description.

To reach the page edit modal, click on “Pages” in the top left menu, locate the “Summer Collection 2024” draft page, and select the “Edit” option:

Navigate to the “SEO” tab, where you will see that both the “Title” and “Description” fields are empty:

Normally, you would have to manually fill in those fields, but no worries—your AI-powered SEO Assistant is here to help!

Click on the button to access the SEO Assistant options. Select “Make a suggestion based on page content” to instruct AI to generate the selected meta field using the content found on the page:

Now you have three options:

1. Use this suggestion: Populate the field with the AI-generated text.

2. Try again: Request an alternative suggestion on the fly.

3. Edit prompt: Manually configure the prompt used to query the AI for SEO meta field generation.

Similarly, you can generate an SEO meta description with a couple of clicks:

Note: The above example focused on a page, but SEO Assistant also works for individual content pieces. Plus, it can be configured to produce results of a given minimum length.

Amazing! SEO Assistant generated a captivating meta title and description for you in just a fraction of a second. It only remains to click “Publish.”

As shown here, using AI for SEO simplifies metadata optimization and removes the tediousness associated with the task.

Integrating AI for Translation in your CMS

Localizing a website into several languages is critical to reaching a global audience. As of 2023, approximately 1.46 billion people are estimated to speak English worldwide. That represents about 18.07% of the global population—less than 1 in 5 people.

As you can imagine, localization becomes essential when targeting local markets in regions such as Asia, Europe, or Latin America. Thankfully, Apostrophe comes with comprehensive support for localization.

When translating a website, a common practice is to start with layout elements like menus, footers, and headers. Although this initial step can be time-consuming, it only needs to be done once as those elements are shared across all pages. That might create the misconception that translating a website into a new language is a quick task. Well, that is not true!

The real challenge arises when realizing that you have to translate all pages, content, and media files. This process can take up to months, especially on large sites. Luckily, you can now automate the task by using AI for translation.

Let’s see how the Apostrophe Pro Automatic Translation extension can help you translate pages and pieces with just a few clicks!

Automatic Translation in Action

Follow the integration guide to install and set up the Automatic Translation extension in your Apostrophe application.

Automatic Translation comes with built-in support for Google Cloud Translation and DeepL as AI translation providers (with support for Azure AI Translator coming soon). To integrate them, you just need a valid API key. Keep in mind that the extension also supports custom providers.

In this example, we will use DeepL, but any other AI translation provider will do.

Suppose you already configured the es locale in your Apostrophe e-commerce project and translated all layout elements. The next step is to translate all content from English to Spanish, which will make your site more accessible to the 500 million native Spanish speakers worldwide. ¡Empezamos!

This is what the English version of the e-commerce site currently looks like:

The goal is to have the entire site translated into Spanish.

The first document to translate is undoubtedly the home page. To do that, click on "Pages" in the top left menu, locate the "Home" page, and select the "Localize…" option:

This will open the Automatic Translation modal below:

A wizard will guide you through the process of configuring how Apostrophe will use AI for translation. Specifically, you will be asked to:

Select the content to localize. The available options are the current document, the current document and its related documents (such as images and any other referenced documents), or only the related documents.
Choose the locales to translate the content into.
Specify the related document types to localize, if you chose to translate them in step 1. You can also decide whether to translate only new related documents or to overwrite existing translated documents. Select the “Translate text content option“ for automatic AI translation.

The home page depends on some related documents. Therefore, the recommended option is "This document and related documents":

In the last step of the wizard, you can select which types of related documents you want to localize. Apostrophe will automatically translate them together with the current document.

The following notifications will inform you that the current document and the selected related documents were translated successfully:

Visit the home page of the Spanish version of your e-commerce site, and you will now see:

Wow, a good start! However, notice that there are no products or categories on the page. This is because they need to be translated as well.

Before translating them, you may find that you are not satisfied with the translation results provided by the AI. No problem, you can always edit the localized documents using Apostrophe's editing capabilities:

The translated pages will be saved as drafts in the selected locale(s). This way, a content manager will have to manually review and approve them for publication. Also, Apostrophe will automatically mark the AI-translated fields, as they require additional attention from your editors.

Also, keep in mind that Automatic Translation also translates the SEO meta title and description into the target language. To verify that, explore the “SEO” tab of the “Summer Collection 2024” page mentioned in the previous chapter:

Observe how both the meta title and the SEO description were translated into Spanish.

Awesome, you are ready to translate products and categories!

Click on "Content" in the top left menu, select a content category, and follow the same procedure as above to translate each piece:

Considering that pieces often involve or are connected to other documents, you can choose the “this document and related documents” option to speed up the translation process.

Use AI to translate all pages and pieces in your project, and this is the end result you will get in just a few minutes:

While it may not be perfect, this is definitely impressive!

Similarly, this is what an AI-translated page product looks like:

Note: Automatic Translation offers a wide range of options and customizations to tailor the AI content translation experience to your needs. For more details, see the developer documentation.

As proven here, leveraging AI for translation significantly accelerates the content localization process. That creates a tremendous opportunity to make your site accessible to millions of users worldwide.

Benefits of SEO Assistant and Automatic Translation Extensions

A few months ago, it was merely a guess, but now there is little doubt that integrating AI directly into a CMS introduces substantial benefits. Specifically, using AI for SEO and translation provides these advantages:

Expanding content reach: AI-driven SEO and translation help to reach a broader audience. Enhanced SEO rankings increase your site's visibility, while localized content resonates with diverse audiences in different regions, significantly boosting the chances of attracting new users to your pages.
Improving productivity: Tasks that once required hours of meticulous work can now be completed in seconds. AI handles the heavy lifting, leaving you to refine and personalize the generated content. This shift reduces time spent on manual tasks and helps you concentrate on what truly matters, such as creating outstanding user experiences.
Saving money: By delegating specialized tasks like SEO optimization and translation to AI, you may no longer need to hire a team of dedicated experts in those fields. Such a cost-effective strategy enables you to allocate resources more efficiently while still achieving high-quality results, ultimately saving your business money.

The above improvements are made possible by SEO Assistant and Automatic Translation, the new AI-powered Apostrophe Pro extensions.

These are just a few of the several powerful features available in the Pro and Assembly tiers of Apostrophe, which also include advanced permissions, document version history, and more.

Conclusion

In this article, we looked at what the new AI-powered SEO Assistant and Automatic Translation extensions have to offer. With AI integrated directly into Apostrophe, you can now generate SEO metadata and translate content into any language within seconds and a handful of clicks.

While the current results are already spectacular, this is just the beginning. The Apostrophe team is continually working to discover new and effective AI integrations, and you should be excited about what the future holds.

To get a preview of what's next for Apostrophe, keep an eye on our product roadmap!

An Introduction to Unit Testing in Node.js

Antonello Zanini — Thu, 22 Aug 2024 15:13:50 +0000

Unit tests are essential to verify the behavior of small code units in a Node.js application. This leads to clearer design, fewer bugs, and better adherence to business requirements. That's why Test-Driven Development (TDD) and Behavior-Driven Development (BDD) have become so popular in the backend development community.

In this tutorial, we'll dive into unit testing and understand why it's needed in your backend. We'll then learn how a unit test is structured and explore the most widely used libraries for unit testing in Node.js.

It's time to become a Node.js unit testing expert!

What Is Node.js Unit Testing?

Unit testing is a software testing practice in which individual units of an application are tested to ensure they behave as expected. In particular, a unit test is a short and isolated test that verifies the correctness of a specific piece of code — typically a function or method.

Thus, Node.js unit testing involves writing tests to validate the functionality of individual modules in your backend. This includes testing API endpoints, database interactions, and business logic functions to ensure robustness.

A CI/CD pipeline usually runs unit tests to verify the integrity of a deployment, ensuring that a Node.js application stays reliable, even after code changes are made pre-production.

Why Write Unit Tests in Node.js?

Now that you know what unit testing is in Node.js, you may wonder why your backend needs unit tests.
Here are three good reasons:

To discover bugs earlier: Unit tests allow you to detect bugs in APIs during development or in the CI/CD deployment pipeline. By identifying issues earlier, you can fix them immediately and reduce the likelihood of runtime errors. Thanks to higher code coverage, your users will have a better experience.
To increase code quality: Unit testing serves as a form of documentation for your code. That's because each Node.js unit test provides a clear example of how a specific function should behave. By writing tests that cover different scenarios, you can verify that your backend is resilient to unexpected inputs. This leads to higher code quality and increased maintainability.
To improve your architecture design: To make the codebase of your backend testable, you must organize your codebase accordingly. This encourages designing a modular and decoupled Node.js architecture. As a rule of thumb, you don't want to write complex components that are difficult to test.

What a Unit Test in Node.js Looks Like

Most Node.js test suites follow the BDD specification style, which revolves around the describe() and it() functions:

describe("user service functions", () => {
  it("should retrieve the user with the given ID", async () => {
    // ...
  });

  it("should update the information of an existing user", async () => {
    // ...
  });

  // other it() unit tests...
});

In this convention, a single JavaScript test file typically contains multiple unit tests represented as it() functions. These test functions can then be encapsulated within a describe() block for clarity and logical organization.

Let's now explore the logical sections a Node.js unit test usually consists of.

1. Setup

Here, you focus on preparing an environment to run unit tests in isolation. In a Node.js application, this involves setting up the right environment variables and a sample database. This step aims to ensure that tests produce predictable and consistent results.

To achieve that, the BDD specification style provides the following two hooks:

before(): Runs once before all the tests in describe().
beforeEach(): Runs before each individual unit test case in describe().

These hooks are useful to initialize resources and shared variables for tests.

To ensure that Node.js unit tests run in isolation, you may also need to rely on the following techniques:

Mocking: Creating fake objects to simulate external dependencies that the functions you're testing depend on.
Stubbing: Overriding specific functions or methods with default behavior.
Spying: Observing function calls to track their behavior during testing.

2. Testing

Testing directly executes the lines of code you want to test to exercise the functionality you're testing and observe its behavior.
Specific lines of code — commonly business logic functions — are called to retrieve data or perform operations. In other words, you provide inputs to a function and retrieve the resulting output.

3. Assertion

Assertions are predicates that make sure your code behaves as intended. The results you obtain after testing are compared with the expected results.

Node.js test suite libraries come with an assertion engine that makes it easier to write more expressive and readable assertions. You get functions to compare values and verify conditions in a structured way.

This step is critical to validate the correctness of the code being tested. It checks whether the tested code produces the correct results on the given inputs and conditions.

4. Cleanup

This (optional) phase of a Node.js unit test releases the allocated resources from the preparation step. The BDD specification style provides two hooks for cleanup:

after(): Runs once after all unit tests in describe() have been executed.
afterEach(): Runs after each individual unit test defined in describe().

The code in these hooks does tasks like closing database connections, deleting temporary files, and resetting temporary settings.

Popular Libraries to Perform Unit Testing in Node.js

Suppose you have a sample Node.js endpoint that returns the Fibonacci series up to n terms, where n is a query parameter. This is what your Express server may look like:

// server.js
const express = require("express");
const { generateFibonacciSeries } = require("./services/math");

const app = express();
const PORT = process.env.PORT || 3000;

app.get("/api/v1/math/fibonacci/:n", (req, res) => {
  const n = parseInt(req.params.n);

  const fibonacciSeries = generateFibonacciSeries(n);

  res.json({ fibonacciSeries: fibonacciSeries });
});

app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

The business logic to calculate the Fibonacci series is encapsulated in the following function:

// services/math.js

function generateFibonacciSeries(n) {
  const fibonacciSeries = [0, 1];
  for (let i = 2; i < n; i++) {
    fibonacciSeries.push(fibonacciSeries[i - 1] + fibonacciSeries[i - 2]);
  }
  return fibonacciSeries;
}

module.exports = {
  generateFibonacciSeries,
};

Now it's time to explore the most popular Node.js unit testing frameworks and libraries using this example. We'll take a look at each tool, its pros and cons, and how to use it to test the above Node.js business logic function.

Mocha

Mocha is a simple and flexible JavaScript testing framework for browser and Node.js applications. Unlike other testing frameworks, it takes a minimalist approach and relies on external libraries for key tasks. It uses Sinon for handling spies, stubs, and mocks, and Chai as the assertion engine. Mocha is extensible through many plugins and can integrate with most test runners.

This is how you can use Mocha to write a Node.js unit test for generateFibonacciSeries():

const assert = require("assert");
const { generateFibonacciSeries } = require("../services/math");

describe("verify that math service works", function () {
  it("the Fibonacci series should return [0, 1, 1, 2, 3, 5] when n is 6", function () {
    const expectedFibonacciSeries = [0, 1, 1, 2, 3, 5];
    const actualFibonacciSeries = generateFibonacciSeries(6);
    assert.deepEqual(actualFibonacciSeries, expectedFibonacciSeries);
  });
});

📈 npm weekly downloads: Over 6.8 million

⭐ GitHub stars: 22.4k+

👍 Pros:

Simple setup and lightweight approach to unit testing.
Highly extensible via plugins for assertions, reporters, and mock libraries.
Great for testing asynchronous operations.
Chai's assertion API is extremely rich.

👎 Cons:

Configuration overhead when setting up different assertion libraries, mocking tools, and other plugins.

Jest

Jest is an all-in-one JavaScript testing framework with a focus on simplicity. It supports projects using Babel, TypeScript, Node.js, React, Angular, Vue, and more. To maximize performance, it runs tests in parallel on isolated processes. Jest comes with an integrated assertion engine based on the expect() function and requires zero configuration for most projects.

Here's how to use Jest to write a unit test for generateFibonacciSeries():

const { generateFibonacciSeries } = require("../src/services/math");

describe("verify that math service works", () => {
  it("should return [0, 1, 1, 2, 3, 5] for Fibonacci series when n is 6", () => {
    const expectedFibonacciSeries = [0, 1, 1, 2, 3, 5];
    const actualFibonacciSeries = generateFibonacciSeries(6);
    expect(actualFibonacciSeries).toEqual(expectedFibonacciSeries);
  });
});

📈 npm weekly downloads: Over 20.2 million

⭐ GitHub stars: 43.5k+

👍 Pros:

Ready to go after installation, with no configuration overhead.
Comes with built-in mocking, snapshot testing, and code coverage out of the box.
Optimized for speed with parallel test execution.
Well-documented, with a large and active community providing support.

👎 Cons:

Installs many dependencies during the initial setup.

Jasmine

Jasmine is a BDD testing framework for Node.js projects and other JavaScript applications. The library comes with no external dependencies, so it is simple to use with a low overhead. It's the oldest of the three tools we've covered, having been around since 2010. Jasmine runs on any JavaScript platform and is compatible with other testing libraries.

See Jasmine in action when writing a Node.js unit test for generateFibonacciSeries():

const { generateFibonacciSeries } = require("../src/services/math");

describe("verify that math service works", () => {
  it("should return [0, 1, 1, 2, 3, 5] for Fibonacci series when n is 6", () => {
    const expectedFibonacciSeries = [0, 1, 1, 2, 3, 5];
    const actualFibonacciSeries = generateFibonacciSeries(6);
    expect(actualFibonacciSeries).toEqual(expectedFibonacciSeries);
  });
});

Note that the expect() functions of Jest and Jasmine have a similar syntax, but they are not the same function.

📈 npm weekly downloads: Over 3.8 million

⭐ GitHub stars: 15.7k+

👍 Pros:

Offers built-in support for assertions, mocks, and spies.
Provides a readable and expressive syntax for writing tests.
Works in both browser and Node.js environments with easy configuration.
Thoroughly tested and documented over years of development.

👎 Cons:

Requires a separate test runner like Karma or Jasmine CLI for executing tests.

Node.js Unit Testing: Best Practices

Here's a list of best practices to build robust Node.js unit tests:

Keep unit tests small and focused on single functions to make them easier to understand and maintain.
Avoid declaring more than one assertion in the same test.
Use descriptive test names that clearly indicate what is being tested and the expected outcome.
Rely on mocks or stubs to isolate the code that's being tested from external dependencies such as databases, APIs, or other modules.
Use helper functions, setup functions, and utilities to avoid duplicating test code.
Integrate your unit tests into your CI/CD pipeline to ensure that tests run automatically at each deployment.
Adopt code coverage tools to monitor test coverage and identify areas of the codebase that are not adequately tested.
Write clear and concise documentation for your tests, including information about what is being tested, why it's important, and any relevant context or assumptions.

Wrapping Up

In this blog post, we explored Node.js unit testing and its benefits for backend projects.

You now know:

The definition of unit tests.
What elements a Node.js unit test consist of.
The most popular unit testing libraries and frameworks for Node.js.

Thanks for reading!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

How to Define a Counter in MySQL

Antonello Zanini — Fri, 09 Aug 2024 13:40:45 +0000

The simplest way to identify an object in a database is to assign it a unique integer, as it happens with order numbers. Not surprisingly, most databases support the definition of auto-incremental values. Essentially, an incremental value is just a counter used to uniquely identify an entry in a table. Well, there are several ways to define a counter in MySQL!

In this article, you will understand what a counter is, where it is useful in a database, and how to implement it in MySQL.

Let's dive in!

What Is Counter?

In programming, a counter is a variable used to keep track of the number of occurrences of certain events or actions. In most cases, it is used in a loop to increment a numerical value automatically and keep track of the number of iterations. This is why counters are generally associated with the concept of auto-incremental numbers.

In databases, a counter is commonly used to generate unique identifiers for records, such as sequential numbers for primary keys or tracking numbers for events.

MySQL provides the AUTO_INCREMENT attribute to automatically increase a column value by one with each new record. However, this attribute only increments values by one unit at a time. For more customized behavior, you might need to manually define a MySQL counter.

Reasons to Use a Counter in MySQL

These are the top 5 reasons why using a counter in MySQL:

Create unique identifiers: Sequential values are ideal for primary keys to ensure data integrity and consistency.
Enhanced data management: Automatically incremented counters help to sort and identify records.
Simplified data entry: Automatic unique ID generation reduces manual entry errors and simplifies the data insertion process.
Efficient record tracking: Counters make it easier to track and manage records. That is particularly true in applications where sequential or unique numbering is critical, such as order processing or inventory management.
Customizable formats: By using counters with custom formats, you can create meaningful identifiers that provide context and organization to your data.

Defining a Counter in MySQL

Explore the two most common approaches to defining counters in MySQL.

Note: The MySQL sample queries below will be executed in DbVisualizer, the database client with the highest user satisfaction in the market. Keep in mind that you can run them in any other SQL client.

Using AUTO_INCREMENT

By marking a column with the AUTO_INCREMENT attribute, MySQL will automatically give it an incremental value when inserting new records. This ensures that each entry has a unique, sequential identifier.

Consider the following example:

CREATE TABLE employees (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(50),
    surname VARCHAR(50),
    email VARCHAR(100),
    role VARCHAR(50)
);

The above query creates an employees table with an AUTO_INCREMENT primary key id.

Now, suppose the employees table already contains these 5 records:

To add three more records, use the following INSERT query:

INSERT INTO employees (name, surname, email, role) VALUES
('Frank', 'Miller', 'frank.miller@example.com', 'Developer'),
('Grace', 'Davis', 'grace.davis@example.com', 'Manager'),
('Hank', 'Wilson', 'hank.wilson@example.com', 'Designer');

employees will now contain:

Note that the id column of the new records has been populated with an incremental value by default. In particular, you can omit the AUTO_INCREMENT column in the INSERT statement (or set it to NULL), as MySQL will populate it for you.

Note that the AUTO_INCREMENT attribute only works on integer PRIMARY KEYs. Additionally, AUTO_INCREMENT values are always incremented by one unit at a time. Check out this guide to find out more about numeric data types in MySQL.

To customize the behavior of AUTO_INCREMENT, you can use the following variables:

auto_increment_increment: Defines the increment step for the AUTO_INCREMENT values. The default value is 1.
auto_increment_offset: Sets the starting point for the AUTO_INCREMENT values. For example, setting it to 5 will make the first AUTO_INCREMENT value start from 5.

At the same time, these variables apply to all AUTO_INCREMENT columns in the database and have either global or session scope. In other words, they cannot be applied to individual tables.

The following approaches will give you more flexibility when defining counters in MySQL.

Using a Variable

A simple approach to creating a custom counter in MySQL is to use a user-defined variable.

Now, suppose you want each employee to have an internal auto-incremental ID in the following format:

Roogler#<incremental_number>

Add an internal_id column to employees:

ALTER TABLE employees
ADD COLUMN internal_id VARCHAR(50);

Then, you can achieve the desired result with the following query:

-- initialize the counter
SET @counter = 0;

-- update the internal_id column with the formatted incremental values
UPDATE employees
SET internal_id = CONCAT('Roogler#', @counter := @counter + 1);

This uses a variable to implement the counter and the CONCAT function to produce the internal ID in the desired format.

Note that the starting value and the way you increment the counter are totally customizable. This time, there are no restrictions.

Execute the query in your MySQL database client:

Note that DbVisualizer comes with full support from MySQL variables.

If you inspect the data in the employees table, you will now see:

Wonderful! Mission complete.

The main drawback of this solution is that user-defined variables in MySQL are session-specific. This means that their values are only retained for the duration of the current session. So, they are not persistent across different sessions or connections.

For a more persistent solution, you could use a stored procedure along with an SQL trigger to automatically update the internal_id every time a new employee is inserted. For detailed instructions, see this article on how to use stored procedures in SQL.

Conclusion

In this guide, you saw what a counter is, why it is useful, and how to implement it in MySQL. You now know that MySQL provides the AUTO_INCREMENT keyword to define incremental integer primary keys and also supports custom counter definition.

As learned here, dealing with auto-incremental values becomes easier with a powerful client tool like DbVisualizer. This comprehensive database client supports several DBMS technologies, has advanced query optimization capabilities, and can generate ERD-type schemas with a single click. Try DbVisualizer for free!

FAQ

How to count records in MySQL?

To count records in MySQL, use the COUNT aggregate function as in the sample query below:

SELECT COUNT(*) FROM table_name;

This returns the total number of rows in the specified table. When applied to a column, COUNT(column_name) counts all non-NULL values in the specified column.

What is the difference between COUNT and a counter in MySQL?

In MySQL, COUNT is an aggregate function to calculate the number of rows in a result set. Instead, a counter is a mechanism used to generate sequential numbers. COUNT is used for aggregation and reporting, whereas a counter is employed to assign a unique identifier or track the order of records as they are inserted into a table.

Should I use AUTO_INCREMENT or define a custom counter in MySQL?

Use AUTO_INCREMENT when you need an automatic way to generate sequential IDs for a primary key. On the other hand, if you require custom behavior—like specific formatting, starting values, or incrementing patterns—a custom counter implemented using a variable might be more appropriate.

How to define a variable in MySQL?

To define a variable in MySQL in a session, you must use the SET statement as follows:

SET @variable_name = value;

In this case, @variable_name is the variable and value is its initial value. This variable can be used within the session for calculations, conditions, or as part of queries. For local variables within stored procedures or functions, you need instead the DECLARE statement followed by SET:

DECLARE variable_name datatype;
SET variable_name = value;

Does DbVisualizer support database variables?

Yes, DbVisualizer natively supports more than 50 database technologies, with full support for over 30 of them. The full support includes database variables and many other features. Check out the list of supported databases.

The post "How to Define a Counter in MySQL" appeared first on Writech.

How to Perform Data Validation in Node.js

Antonello Zanini — Wed, 03 Jul 2024 15:02:41 +0000

Data validation is essential to avoid unexpected behavior, prevent errors, and improve security. It can be performed both on a web page — where data is entered — and on the server, where the data is processed.

In this tutorial, we'll explore data validation in the Node.js backend. Then, you'll learn how to implement it in Express using the express-validator library.

Get ready to become a Node.js data validation expert!

What Is Data Validation?

Data validation ensures that data (whether entered or provided) is correct, consistent, and useful for its intended purpose. This process is typically performed in frontend applications, such as when dealing with forms.

Likewise, it is essential to validate data in the backend as well. In this case, data validation involves checking path and query parameters, as well as the body data sent to servers. This ensures that the data received by each API meets the specified criteria, preventing errors and vulnerabilities and ensuring the smooth functionality of your application.

Main Benefits of Validating Incoming Requests in Node.js

Validating incoming requests in Node.js offers several key benefits:

Enhancing security: Mitigate some threats, such as injection attacks and data breaches. Proper validation prevents attackers from exploiting vulnerabilities in your application by sending it malformed or malicious data.
Improving reliability: Ensure that only valid and sanitized data is processed and stored in the backend application. That enhances the overall integrity and reliability of the data, leading to a more robust and trustworthy server.
Maintaining compliance: Make sure that the data handled by the server adheres to specific data format requirements or meets internal coding standards.

Now that you know what data validation is and why you should enforce it in your Node.js application, let's see how to do it in this step-by-step tutorial!

Prerequisites

To follow this tutorial, you need a Node.js 18+ application with a few endpoints. For example, the following Express server is perfect:

const express = require("express");

// initialize an Express server
const app = express();
app.use(express.json());

// an array to use as an in-memory database
const users = [
  { id: 1, email: "john.doe@example.com", fullName: "John Doe", age: 30 },
  { id: 2, email: "jane.smith@example.com", fullName: "Jane Smith", age: 25 },
  { id: 3, email: "bob.johnson@example.com", fullName: "Bob Johnson", age: 40 },
  {
    id: 4,
    email: "alice.williams@example.com",
    fullName: "Alice Williams",
    age: 35,
  },
  { id: 5, email: "mike.brown@example.com", fullName: "Mike Brown", age: 28 },
  { id: 6, email: "sara.taylor@example.com", fullName: "Sara Taylor", age: 33 },
  { id: 7, email: "chris.lee@example.com", fullName: "Chris Lee", age: 22 },
  { id: 8, email: "emily.davis@example.com", fullName: "Emily Davis", age: 45 },
  {
    id: 9,
    email: "alex.johnson@example.com",
    fullName: "Alex Johnson",
    age: 27,
  },
  {
    id: 10,
    email: "lisa.wilson@example.com",
    fullName: "Lisa Wilson",
    age: 38,
  },
];

// define three sample endpoints
app.get("/api/v1/users/:userId", (req, res) => {
  const userId = req.params.userId;
  // find a user by id
  const user = users.find((user) => user.id == userId);

  if (!user) {
    res.status(404).send("User not found!");
  } else {
    res.send({
      user: user,
    });
  }
});

app.get("/api/v1/users", (req, res) => {
  // select all users by default
  let filteredUsers = users;

  const search = req.query.search;
  if (search !== undefined) {
    // filter users by fullName with a case-insensitive search
    filteredUsers = users.filter((user) => {
      return user.fullName.toLowerCase().includes(search.toLowerCase());
    });
  }

  res.send({
    users: filteredUsers,
  });
});

app.post("/api/v1/users", (req, res) => {
  const newUser = req.body;
  const maxId = users.reduce((max, user) => (user.id > max ? user.id : max), 0);

  // add a new user with an auto-incremented id
  users.push({
    id: maxId + 1,
    ...newUser,
  });

  res.status(201).send();
});

// start the server locally on port 3000
const port = 3000;
app.listen(port, () => {
  console.log(`Server listening at http://localhost:${port}`);
});

This defines a local variable named users as an in-memory database. Then, it initializes an Express application with the following three endpoints:

GET /api/v1/users/:userId: To retrieve a single user from the users array based on its id.
GET /api/v1/users: To get the list of users in the database. It accepts an optional search query parameter to filter users by their full name.
POST /api/v1/users: To add a new user to the users array.

Next, you'll need a library to perform data validation in Node.js.
With thousands of weekly downloads, express-validator is the most popular option.

express-validator provides a set of Express middleware to validate and sanitize incoming data to server APIs. Behind the scenes, these middleware functions are powered by validator.js. If you are unfamiliar with this package, validator.js is the most widely used data validation library in the entire JavaScript ecosystem.

What makes express-validator so successful is its rich set of features and intuitive syntax for validating Express endpoints.
It also provides tools for determining whether a request is valid, functions for accessing sanitized data, and more.

Add the express-validator npm package to your project's dependencies with:

npm install express-validator

Perfect! You now have everything you need to perform data validation in an Express application.

For a faster setup, clone the GitHub repository supporting this guide:

git clone https://github.com/Tonel/nodejs-express-validator-demo

You'll find the Express server above and further implementations in dedicated branches.

express-validator supports two equivalent ways to implement data validation:

Validation Chain: Define the validation rules by calling one method after another through method chaining.
Schema Validation: Define the validation rules in an object-based schema to match against the incoming data.

Let's dive into both!

Data Validation in Node.js With Validation Chains

Learn how to implement data validation through validation chains in express-validator.

Understand Validator Chains

In express-validator, validation chains always begin with one of the following middleware functions:

check(): Creates a validation chain for the selected fields in any of the req.body, req.cookies, req.headers, req.query, or req.params locations. If the specified fields are present in more than one location, the validation chain processes all instances of that field's value.
body(): Same as check(), but it only checks fields in req.body.
cookie(): Same as check(), but it only checks fields in req.cookies.
header(): Same as check(), but it only checks fields in req.headers.
param(): Same as check(), but it only checks fields in req.params.
query(): Same as check(), but it only checks fields in req.query.

These middleware functions accept one or more field names to select from incoming data. They also provide some methods, which is possible because JavaScript functions are actually first-class objects. Their methods always return themselves, leading to the method chaining pattern.

So, let's assume that you want to ensure that the name body parameter contains at least 4 characters when it is present.
This is how you can specify that with an express-validator validator chain:

body("name").optional().isLength({ min: 4 });

Each method chain will return a valid Express middleware function you can use for validation in a route handler.
A single route handler can have one or more validation middleware, each referring to different data fields.

Check out the express-validator documentation for all validation chain methods.

Time to see validation chains in action!

Validate Route Parameters

Suppose you want to ensure that the userId parameter in GET /api/v1/users/:userId is an integer. This is what you may end up writing:

app.get("/api/v1/users/:userId", param("userId").isInt(), (req, res) => {
  // business logic...
});

While the validation chain defined above is correct, it's not enough, as express-validator doesn't report validation errors to users automatically. Why? Because it's better if developers always manually define how to handle invalid data!

You can access the result object of data validation in an Express endpoint through the validationResult() function.

Import it along with the validation middleware function from express-validator:

const {
  check,
  body,
  // ...
  validationResult,
} = require("express-validator");

Then, you can define the validated route handler for GET /api/v1/users/:userId as below:

app.get("/api/v1/users/:userId", param("userId").isInt(), (req, res) => {
  // extract the data validation result
  const result = validationResult(req);
  if (result.isEmpty()) {
    const userId = req.params.userId;
    // find a user by id
    const user = users.find((user) => user.id == userId);

    if (!user) {
      res.status(404).send("User not found!");
    } else {
      res.send({
        user: user,
      });
    }
  } else {
    res.status(400).send({ errors: result.array() });
  }
});

When userId is not an integer, the endpoint will return a 400 response with validation error messages. In production, you should override that response with a generic message to avoid providing useful information to potential attackers.

Validate Query Parameters

In this case, you want to ensure that the search query parameter in the GET /api/v1/users endpoint is not empty or blank when it is present.

This is the validation chain you need to define:

query("search").optional().trim().notEmpty();

Note that the order of the method calls in the chain is important. If you switch trim() with notEmpty(), blank strings will pass the validation rule.

trim() is a sanitizer method and it affects the values stored in search.

If you need to access the sanitized data directly in the route handler, you can call the matchedData() function. The validated and sanitized endpoint will now be specified as follows:

app.get(
  "/api/v1/users",
  query("search").optional().trim().notEmpty(),
  (req, res) => {
    // extract the data validation result
    const result = validationResult(req);
    if (result.isEmpty()) {
      // select all users by default
      let filteredUsers = users;

      // read the matched query data from "req"
      const data = matchedData(req);
      const search = data.search;
      if (search !== undefined) {
        // filter users by fullName with a case-insensitive search
        filteredUsers = users.filter((user) => {
          return user.fullName.toLowerCase().includes(search.toLowerCase());
        });
      }

      res.send({
        users: filteredUsers,
      });
    } else {
      res.status(400).send({ errors: result.array() });
    }
  }
);

Notice that req.query.search will contain the original value of the search query parameter, but you want to use its sanitized value.

Validate Body Data

Now, suppose you want the body of POST /api/v1/users to follow these rules:

fullName must not be empty or blank.
email must be a valid email. If there's an invalid email, the validation error message should be “Not a valid e-mail address.”
age must be an integer greater than or equal to 18.

This is how you can implement the desired data validation:

app.post(
  "/api/v1/users",
  body("fullName").trim().notEmpty(),
  body("email").isEmail().withMessage("Not a valid e-mail address"),
  body("age").isInt({ min: 18 }),
  (req, res) => {
    // extract the data validation result
    const result = validationResult(req);
    if (result.isEmpty()) {
      // read the matched body data from "req"
      const newUser = matchedData(req);
      const maxId = users.reduce(
        (max, user) => (user.id > max ? user.id : max),
        0
      );

      // add a new user with an auto-incremented id
      users.push({
        id: maxId + 1,
        ...newUser,
      });

      res.status(201).send();
    } else {
      res.status(400).send({ errors: result.array() });
    }
  }
);

There are a couple of critical aspects to emphasize in this example. First, a single route handler can have multiple validation middlewares referring to the same data source. Second, the methods offered by the middleware functions not only specify how to validate the data, but also allow you to customize error messages and more.

Test the Validated Endpoints

Launch your Node.js application and verify that the data validation logic works as expected. Otherwise, check out the chain-validation branch from the repository that supports this article:

git checkout chain-validation

Install the project dependencies and launch an Express development server:

npm install
npm run start

Your Node.js application should now be listening locally on port 3000. Open your favorite HTTP client and try to make a GET request to /api/v1/users/edw:

Since edw is not a number, you'll get a 400 response.
Specifically, the error array generated by express-validator contains one or more error objects in the following format:

{
    "type": "field",
    "value": "edw",
    "msg": "Invalid value",
    "path": "userId",
    "location": "params"
}

You can simulate another validation error by calling the GET /api/v1/users endpoint with a blank search query parameter:

Again, trigger a validation error by calling the POST /api/v1/users API with an invalid body:

If you call the three endpoints with the expected data instead, they will return a successful response as expected.

Great, you just learned how to perform data validation in Node.js! All that remains is to explore the equivalent schema-based approach to data validation.

Data Validation in Node.js with Schema Validation

Let's now see how to define data validation through schema objects with express-validator.

Understand Schema Validation

In express-validator, schemas are an object-based way of defining validation and/or sanitization rules on a request.

While their syntax differs from validation chains, they offer exactly the same functionality. Under the hood, express-validator translates schemas into validation chain functions.
Thus, you can choose between one syntax or the other, depending on your preference.
The same Express application can have some endpoints validated with chains and others validated with schemas.

Schemas are simple JavaScript objects whose keys represent the fields to validate. Schema values contain validation rules in the form of objects.
Pass a schema object to the checkSchema() function to get an Express validation middleware.

For example, this is how you can use a schema to ensure that the name body parameter contains at least 4 characters when it is present:

checkSchema(
  {
    name: {
      optional: true,
      isLength: { options: { min: 4 } },
    },
  },
  ["body"]
);

By default, checkSchema() behaves like check(). To specify which input data sources it should check, you can pass them in an array as the second parameter. In the example above, the validation schema object will only be applied to the body data.

Sometimes, you may need to validate a body's inner field. Check out the documentation to see what express-validator offers in field selection.

Validate Route, Query, and Body Data With Schemas

Here's how you can specify the validation rules shown in the method chaining section above through schema validation:

GET /api/v1/users/:userId:

app.get(
  "/api/v1/users/:userId",
  checkSchema(
    {
      userId: { isInt: true },
    },
    ["params"]
  ),
  (req, res) => {
    // extract the data validation result
    const result = validationResult(req);
    if (result.isEmpty()) {
      const userId = req.params.userId;
      // find a user by id
      const user = users.find((user) => user.id == userId);

      if (!user) {
        res.status(404).send("User not found!");
      } else {
        res.send({
          user: user,
        });
      }
    } else {
      res.status(400).send({ errors: result.array() });
    }
  }
);

GET /api/v1/users:

app.get(
  "/api/v1/users",
  checkSchema(
    {
      search: { optional: true, trim: true, notEmpty: true },
    },
    ["query"]
  ),
  (req, res) => {
    // extract the data validation result
    const result = validationResult(req);
    if (result.isEmpty()) {
      // select all users by default
      let filteredUsers = users;

      // read the matched query data from "req"
      const data = matchedData(req);
      const search = data.search;
      if (search !== undefined) {
        // filter users by fullName with a case-insensitive search
        filteredUsers = users.filter((user) => {
          return user.fullName.toLowerCase().includes(search.toLowerCase());
        });
      }

      res.send({
        users: filteredUsers,
      });
    } else {
      res.status(400).send({ errors: result.array() });
    }
  }
);

Note that the order of the attributes in the schema object matters.
Placing the trim attribute after notEmpty will result in a different validation rule.

POST /api/v1/users:

app.post(
  "/api/v1/users",
  checkSchema(
    {
      fullName: {
        trim: true,
        notEmpty: true,
      },
      email: {
        errorMessage: "Not a valid e-mail address",
        isEmail: true,
      },
      age: {
        isInt: { options: { min: 18 } },
      },
    },
    ["body"]
  ),
  (req, res) => {
    // extract the data validation result
    const result = validationResult(req);
    if (result.isEmpty()) {
      // read the body data from the matched data
      const newUser = matchedData(req);
      const maxId = users.reduce(
        (max, user) => (user.id > max ? user.id : max),
        0
      );

      // add a new user with an auto-incremented id
      users.push({
        id: maxId + 1,
        ...newUser,
      });

      res.status(201).send();
    } else {
      res.status(400).send({ errors: result.array() });
    }
  }
);

As you can see, not much changes from chain validation. The two approaches are completely equivalent.

Test the Validated Endpoints

Start your Express application and prepare to test your schema-based data validation.
Otherwise, check out the schema-validation branch from the repository supporting this article:

git checkout chain-validation

Install the project's dependencies, start the local server, and repeat the API calls made in the method chaining validation section.
You should get the exact same results!

Wrapping Up: Protect Your Node.js Server From Unexpected Incoming Data

In this post, we defined Node.js data validation and explored its benefits for a backend application.

You now know:

The definition of data validation
Why you should check the data received by an endpoint before feeding it to business logic
How to implement data validation in Node.js with two different approaches: method chaining and schema validation

Thanks for reading!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Best PostgreSQL GUI Tools

Antonello Zanini — Sat, 15 Jun 2024 14:26:11 +0000

TL;DR: There are many database GUI tools available, but only a few of them have been designed specifically for PostgreSQL. If you want to manage your PostgreSQL databases effortlessly, you must adopt an advanced PostgreSQL GUI tool. Here, you can find a list of the four best PostgreSQL applications for you.

When the first version of PostgreSQL was released, the only way to access data was to run queries from the command line. As you can imagine, interacting with a database via the command line is something that only skilled users can do. This is because you need to know the PostgreSQL language in depth.

Over time, the needs of end users have evolved for developers, data scientists, and data analysts alike. As a result, several PostgreSQL GUI tools have entered the market, and you can now choose between many PostgreSQL clients.

If you've ever felt limited by your PostgreSQL GUI tool or never used one, this article might shed some light on which one to choose. This article will dig into the four best PostgreSQL GUI tools for developers. But first, let's take a look at why you should be using one and which criteria were considered while making the list.

Why You Need a PostgreSQL GUI Tool

Adopting a PostgreSQL GUI tool can bring many advantages to your data management process:

Visual feedback: Managing data visually in the tool makes everything easier and opens the databases to non-technical members of your team.
Better data management: A GUI tool specifically designed to handle PostgreSQL databases makes data management easier.
Scalability: You can use the same tools to connect to several PostgreSQL databases and manage them all.

Elements of a Good PostgreSQL GUI Tool

There are several aspects to consider when selecting and evaluating PostgreSQL GUI tools:

Most commonly used: The more people who adopt the tool, the more documentation there will be online. After all, it's easier to get support if a tool is backed by a large community. This is more likely to happen if the tool is available on all major operating systems.
Easy to use and learn: PostgreSQL is a complex technology, and this is why a GUI tool should make everything easier. A good PostgreSQL GUI tool should be easy to learn and should not require several hours of training.
Plugin support: The ability to extend the PostgreSQL GUI tool with plugins developed by the community makes it future-ready, as new features can be introduced into the tool at any time.
Customizable user interface: You should be able to customize the tool to fit your particular needs. The more customizable the user interface of a PostgreSQL GUI tool is, the better the resulting user experience will be.
Fast and reliable: The tool should be free of major bugs and ensure good performance without using too many resources. Non-technical team members may not be able to run a PostgreSQL GUI tool that requires several GB of RAM because they typically don't have hardware as powerful as technical team members.

Conclusion

A PostgreSQL client is an essential tool when it comes to data management. PostgreSQL clients are particularly important because they make databases simpler for non-engineer members of your team. Data is easier to use and understand, allowing your engineers to save time while supporting the team.

This article explained why you need a PostgreSQL GUI tool and how to identify a good one. You learned about four of the best PostgreSQL GUI tools on the market. TablePlus, OmniDB, and DbVisualizer support other database technologies and have excellent features to support PostgreSQL-based data management processes, while pgAdmin is the only tool officially supported by PostgreSQL developers. Keep in mind that the tool that suits you the best depends on your use cases. So, there is no real winner.

The post "Best PostgreSQL GUI Tools" appeared first on Writech.