DEV Community: Dushmanta

Node.js Multi-Threading with Worker Threads

Dushmanta — Tue, 23 Sep 2025 18:07:33 +0000

Have you ever wondered how JavaScript/Node.js efficiently handles complex requests despite being single-threaded? What happens when a CPU-intensive task runs on your backend, affecting other requests? Is there a way to run these tasks separately and simultaneously? In this article, we'll explore how to achieve this by understanding single-threading vs. multi-threading and how the Worker Threads module can help improve Node.js performance.

SIngle-threaded JavaScript:

We all know that, unlike languages like C++, C#, and Java, which support multi-threading, JavaScript is single-threaded because it was originally built for client-side scripting to make web pages interactive and run only in the browser. But over time, we can now run JavaScript on the server thanks to Node.js. Due to this single-threaded nature, Node.js is very efficient for I/O-bound tasks.

Node.js's non-blocking I/O model has an event loop at its core, allowing Node.js to perform non-blocking operations even though JavaScript is single-threaded. The event loop enables Node.js to handle multiple concurrent operations efficiently, despite its single-threaded nature.

Why Multi-Threading is Important in Node.js:

Now, when dealing with CPU-heavy tasks, the single-threaded nature of Node.js can struggle, causing performance issues. For example, if the main thread handles a time-consuming task like file compression or heavy computation, simple HTTPS requests can take too long to respond. This creates performance issues for the user, which can be very frustrating. To resolve these issues and improve performance, multi-threading can be a great solution.

Before diving into how to achieve multi-threading, let's first understand single-threading and multi-threading and their differences.

Single vs. Multi-threading:

Single-threading refers to executing one task at a time on a single core. This means that only one operation can be processed sequentially at any given moment. For example, if you have two tasks to complete, the second task must wait for the first task to finish, regardless of how long the first task takes. This approach can lead to blocking, where the current task being processed delays the execution of subsequent tasks until it is completed.

On the other hand, multi-threading refers to executing multiple operations or tasks simultaneously at any given moment. In this method, instead of waiting for one process to complete, multiple tasks are executed concurrently by dividing them into multiple threads. This allows tasks to be processed at the same time, improving efficiency and performance.

Example:

Imagine a restaurant with only one chef in the kitchen who can prepare only one dish at a time. When orders come in, they pile up, and it takes a long time to complete them. Customers have to wait a long time to get their orders, which can become a major issue and slow down the entire restaurant.

Imagine having multiple chefs in the restaurant, each handling different dishes at the same time. This helps orders get prepared faster and serves customers more quickly, improving the overall performance of the restaurant.

Similarly, in an application, single-threading can lead to delays and performance issues, whereas multi-threading can achieve better performance with the help of concurrency and parallelism.

How Multi-Threading Works on Single-Core vs Multi-Core CPUs:

Single-core:

One question you might have is whether multi-threading is possible on a single-core CPU, given that single-threading refers to executing one task at a time on a single core. The answer is yes, but it does not achieve true parallelism. However, it still provides better performance than single-threading.

Too overwhelming? Let me explain. On a multi-core system, each thread runs on a single core. So, if you have 2 processes, in multi-threading, two threads will run these processes in parallel on 2 different cores. Now you might wonder, what if you have 10 processes to run but only 8 cores available? Or you have a single core but want to execute two threads? How will this be managed?

Well, there are a few important methods through which these results are achieved. Let's discuss some of them briefly.

Time slicing: In this method, each thread gets a small, equal slice of CPU time in rotation, executing the threads concurrently. The CPU rapidly switches between different tasks, giving each a small slice of processing time. This creates the illusion of parallel processing, even though only one task is running at any given moment.
Context switching: In this method, the CPU rapidly switches between processes, saving the state of the current execution and loading the next. This is repeated to provide concurrent execution.
Priority-based scheduling: Priorities are assigned to threads, with higher-priority threads getting more time and being executed first. This is helpful for critical tasks that need to be prioritized.

Multi-core:

The above methods for a single core work well when the number of threads is greater than the CPU cores. Let's discuss some additional methods:

Load-balancing: The operating system distributes tasks among multiple CPU cores based on the current load. This ensures that all the cores are used effectively and efficiently.
Thread pools: The number of active threads is kept to match the number of cores, and the rest are queued and executed as soon as a core becomes available.
Work Stealing: In this method, idle cores will take tasks from the queues of busy cores, increasing the efficiency of running tasks.

There are also several other methods to use multi-threading in single-core or multi-core systems, but the ones mentioned above are the most commonly used. It's worth mentioning that almost every modern CPU has multiple cores, making the use of multi-core threading increasingly relevant.

Now I've mentioned a few times that running on a single core isn't true parallelism; it offers concurrency. So next, let's understand the difference between concurrency and parallelism.

Concurrency vs. Parallelism:

"Concurrency is about dealing with lot of things at once, Parallelism is about doing a lot of things at once" - Rob Pike

As Rob Pike said, concurrency is when you're dealing with multiple things at once. For example, imagine a chef cooking multiple dishes. The chef might switch between tasks, like cutting vegetables while rice is cooking, then frying chicken while waiting for the dal to finish. When executing multiple threads on a single core, the CPU switches rapidly between threads, handling multiple tasks at once using the methods discussed above.

Parallelism, on the other hand, is doing multiple tasks at the same time. This is like having multiple chefs work on their own dish independently, which is true parallelism. True parallelism is achievable on multi-core CPUs where each thread runs on its own core.

For more about this, you can watch Concurrency is not Parallelism by Rob Pike

When to Use Multi-Threading in Node.js

As mentioned earlier, Node.js can handle I/O-bound tasks efficiently due to its non-blocking I/O model and the Event Loop. This allows Node.js to perform non-blocking operations effectively. These I/O-bound tasks mainly include fetching data from a database, making API calls, or reading files. Behind the scenes, Node.js uses libuv, a multi-platform support library, to handle asynchronous I/O operations efficiently. You can read more about libuv here.

However, when CPU-intensive tasks, also known as CPU-bound tasks like processing images, performing heavy computations, or running loops, come into play, they take time to execute. This causes the Event Loop to be blocked until the task is completed. As a result, other simple tasks or requests cannot be executed, leading to significant performance issues. This goes against the golden rule of Node.js: Don't block the Event Loop.

To solve the problem of CPU-intensive tasks, we need to use multi-threading. This is where the Node.js worker threads module becomes useful.

Introducing the Node.js Worker Threads Module

The Node.js worker threads module was introduced in Node 10.5.0 and was released as stable in Node v12 LTS.

Before understanding how it works, it's important to understand the concept of processes and threads.

Process vs. Thread:

A process is a running program within the operating system that has its own memory and can execute one task at a time. One process cannot see or access the memory of other processes. On single-core machines, these processes run on a single thread concurrently by rapidly switching between them, as explained above. On multi-core machines, the processes run in parallel with one process on each core. If there are more processes than cores, they will be executed concurrently with the help of each core.

Unlike processes, threads do not have their own memory and reside within the memory of one process. When we create a process, it can have multiple threads within the same memory. These threads can communicate with each other through messages and can also pass data using the process's memory.

Worker Threads:

The worker threads module in Node.js allows the application to offload CPU-intensive tasks to another thread, helping to avoid blocking the main thread. This way, the CPU-intensive task runs in another thread using the same process's memory and can communicate with the main thread through messages, sharing data and improving performance.

Let's create a demo Node.js application with Express to see how to implement this module and how it affects our performance. For example, we will use a for loop running 10 billion times and observe the output.

Implementing Worker Thread Module

To implement the worker_threads module, let's create a simple Express app and demonstrate how we can offload the CPU-intensive task to a separate thread and improve performance.

Start a Node.js project by initializing it with the following command:

npm init

Next, install express to set up the server:

npm i express

Create a file called app.js and set up the Express app to run on port 3000:

import express from 'express';

const app = express();
const PORT = 3000;

app.listen(PORT, (error) => {
  if (!error) console.log(`Server is successfully running on port ${PORT}`);
  else console.log(`Error occurred, server can't start. Details: ${error}`);
});

Add a simple / home route to return a success message:

app.get('/', (_, res) => {
  res.json({ success: true, message: 'Home page' });
});

Now, let’s create the /worker route, which runs a CPU-intensive task (e.g., a for loop that runs 10 billion times):

app.get('/worker', (_, res) => {
  let count = 0;
  for (let index = 0; index < 10000000000; index++) {
    count++;
  }
  res.json({ success: true, message: 'Worker page', value: count });
});

Start the server:

node app.js # or index.js

Using Postman, let's test both endpoints. First, hit the home endpoint and see the results:

The / home route responds quickly.

The /worker route takes around 7 seconds to complete the task. Now let's hit the /worker route first and then the / home route

The worker page took 7 seconds, just like before.

But surprisingly, the home page with just a minimal response took 5 seconds. This is because, since it's running on a single thread, the worker route blocked the execution of the request to the home page.

To solve the blocking issue, we can offload the CPU-intensive task to another thread using Node.js's worker_threads module.

First, let's move the logic to a new file called worker-thread.js.

// worker-thread.js
let count = 0;
for (let index = 0; index < 10000000000; index++) {
  count++;
}

Then, create a worker-thread route and import the Worker class from the worker_threads module.

import { Worker } from 'node:worker_threads';

app.get('/worker-thread', (_, res) => {
  //
});

Now, using the parentPort method of worker_threads, we can send data from the worker-threads.js file to the parent port using the message event, which will be triggered.

import { parentPort } from 'node:worker_threads';

let count = 0;
for (let index = 0; index < 10000000000; index++) {
  count++;
}

parentPort.postMessage(count);

Now, using the Worker class, we can receive the data with the message event by providing the file path to a Worker instance and send the received data in a response as follows:

app.get('/worker-thread', (_, res) => {
  const worker = new Worker('./worker-thread.js');
  worker.on('message', (count) => {
    res.json({ success: true, message: 'Worker threads page', value: count });
  });
});

Now, restart the server and test the endpoints again with Postman. We'll test by running the worker-thread endpoint and then the home page to see if the response is delayed or not.

Response from the home page:

Response from the worker-threads route:

As demonstrated, the worker-thread route no longer blocks the request to the home page, even though it still takes 7 seconds to complete the for loop and send the response. This improvement is possible because the logic is now handled in a separate thread, enabling our program to process tasks in parallel. If you'd like to explore the complete implementation, feel free to visit the full code in the GitHub repository.

By utilizing the worker_threads module, you can easily offload CPU-intensive tasks to separate threads, preventing them from impacting the performance of the main thread. This significantly boosts the efficiency of your application, particularly by avoiding event loop blocking. For more detailed information on worker_threads or other methods of multi_threading modules in Node.js, you can refer to the official documents.

Note:

Parallelism: The worker-threads module makes our application more efficient by improving performance through offloading CPU-intensive tasks to another thread. However, it's worth mentioning that we can only achieve true parallelism if the deployed server has multiple core CPUs. With single-core CPUs, we can achieve concurrency, as mentioned earlier.

Thread Management: It's also important to remember that multi-threading comes with challenges. It can increase code complexity, leading to issues like race conditions, thread synchronization,, and memory sharing. So, while using worker-threads, you should be careful with the implementation.

I hope this article has helped clarify the concept of multi-threading and how you can leverage the worker_threads module to enable multi-threading in Node.js. If you spot any errors or have feedback and suggestions for improvement, I'd love to hear from you. Thank you for reading, and have a wonderful day! :)

Transform PostgreSQL into a Vector Database with pgvector

Dushmanta — Sat, 20 Sep 2025 18:03:47 +0000

Building a RAG application is currently a popular topic, and one key requirement is a vector store or database to store and query embeddings. While there are specialized vector databases available, we can also use the OG PostgreSQL as a vector database thanks to the pgvector extension.

In this article, we'll explore how to integrate with an existing PostgreSQL database. The official repository also explains how to install it, so you can check there for more details.

If you're starting from scratch, there's a Docker image available with built-in vector support, officially maintained by the pgvector community. If you already have a database, it's better to integrate with it since it already contains data.

Prerequisite:

Before starting, ensure you have a container running with PostgreSQL. If it's installed locally, make sure it's running on your system.

Installation:

Find PostgreSQL Container

Since we're using PostgreSQL in a container, let's locate the container with the following command:

docker ps

This will list all the containers running on our system. You need to check the Names column to find the name of the Postgres container. Alternatively, you can list only the containers running from the Postgres image with the following command:

docker ps --filter "ancestor=postgres"

Here are my results for the above command:

CONTAINER ID   IMAGE             COMMAND                  CREATED       STATUS       PORTS                                       NAMES
36622542432f   postgres:latest   "docker-entrypoint.s…"   3 weeks ago   Up 4 hours   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   postgres_container

Now, as you can see, my container name is postgres_container.

Next, let's open the container using a terminal with the following command:

docker exec -it <container-name> bash
# docker exec -it postgres_container bash

Installing `pgvector`

Now, the Docker image is based on an Ubuntu (Debian-based Linux distribution) image. Since we're inside the Docker image, we can run apt commands. We need to install some packages to ensure essential tools are available to build the extension, as well as all necessary packages for Postgres to use the pgvector extension.

You can install the necessary packages with the following command:

apt-get update
apt-get install -y build-essential git postgresql-server-dev-all

Now that these packages are installed, let's proceed with installing the pgvector extension.

First, let's copy the source from the official repository (preferably in the root directory).

git clone https://github.com/pgvector/pgvector.git

Next, install the extension from this source with the following command:

cd pgvector
make
make install

After successfully installing, exit and restart the container using the following command:

docker restart <container-name>
# docker restart postgres_container

Now that the pgvector extension is successfully added to the container, the next step is to enable it for the database which we will integrate with pgvector.

Enable `pgvector`

First, let's connect to the PostgreSQL database instance from our terminal using the following command:

docker exec -it <container-name> psql -h localhost -U <db-user> -d <db-name>

# docker exec -it postgres_container psql -h localhost -U dushmanta -d klansity

After successfully connecting, run the following command to create the extension in the database:

CREATE EXTENSION vector;

This command will create the vector extension in the database we are logged into.

Next, let's double-check if it appears in the extensions using the following command:

\dx

If the vector extension is successfully created, you'll see it in the results. In my case, this is how it appears:

List of installed extensions
  Name   | Version |   Schema   |                     Description                      
---------+---------+------------+------------------------------------------------------
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
 vector  | 0.8.0   | public     | vector data type and ivfflat and hnsw access methods
(2 rows)

After successfully installing, let's check if it works.

Testing

Here, we'll create a table with vector fields. Then, we'll store some vectors, fetch them, and see if it works.

Create Table

We'll create a table named pgvector with a field called embedding, which is a vector field with a dimension of 3, using the following command.

CREATE TABLE pgvector (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding VECTOR(3)
);

If you want to learn more about vector dimensions, I suggest checking out these Cloudflare docs.

Note: Set the dimensions based on the model you’re going to use to generate the vector embedding.

If you have pgAdmin, Adminer, or phpMyAdmin running, check if the database is created. You can also verify this in the terminal with the following command.

\d pgvector

My results looked like this:

                               Table "public.pgvector"
  Column   |   Type    | Collation | Nullable |               Default                
-----------+-----------+-----------+----------+--------------------------------------
 id        | integer   |           | not null | nextval('pgvector_id_seq'::regclass)
 content   | text      |           |          | 
 embedding | vector(3) |           |          | 
Indexes:
    "pgvector_pkey" PRIMARY KEY, btree (id)

And in Adminer, it looks something like this:

Insert Vector Embedding

Now let’s insert a vector embedding into the embedding field using the following command:

INSERT INTO pgvector (content, embedding) VALUES
('first doc', '[1,2,3]'),
('second doc', '[4,5,6]');

This will create two records. Let's check the records.

Note: In real use cases these embedding will be generated by models and the vector numerical values will be very different.

SELECT * FROM pgvector;

Here are the results:

 id |  content   | embedding 
----+------------+-----------
  1 | first doc  | [1,2,3]
  2 | second doc | [4,5,6]
(2 rows)

Vector Search

Now that we've added two records with vector data, let's perform a vector search using the vector [1,2,2] to find which data is closest to it with the following command:

SELECT id, content, embedding
FROM pgvector
ORDER BY embedding <-> '[1,2,2]' -- THe <-> symbol is used for vector search
LIMIT 1;

The result will be the following:

 id |  content  | embedding 
----+-----------+-----------
  1 | first doc | [1,2,3]
(1 row)

As you can see, it fetched the [1,2,3] value as its nearest.

Delete Table

Since we created this table for testing purposes, let’s delete it with the following command.

DROP TABLE pgvector;

Now we have successfully installed the pgvector extension and can use our PostgreSQL database as a vector database for GenAI applications.

This article is intended as a personal note, but if you're looking to integrate pgvector, I hope you find it helpful. If you notice any mistakes or have any issues, please feel free to contact me. Thank you.

Amazon Bedrock with JavaScript: Text, Chat, Streaming & Structured Output via SDK and REST

Dushmanta — Wed, 06 Aug 2025 20:09:17 +0000

As AI becomes more prevalent, integrating Generative AI into applications has become a common trend. Whether it's necessary and solves any problem is another matter.

Different service providers have their own LLMs, while some offer a variety of foundational models in one place with a single API key. Amazon Bedrock is one of these providers. In this article, we’ll explore how to integrate Amazon Bedrock into our JavaScript application. Before we begin, I want to thank Péter Harang for his article on Setting up AWS Bedrock for API-based text inference, which has been very helpful for this implementation.

First, let's understand what Amazon Bedrock is.

What is Amazon Bedrock?

Amazon Bedrock is a fully managed service from AWS that lets you access various foundation models from top providers in one place. This makes it easier to integrate Generative AI into applications. It also allows you to customize with your own dataset for fine-tuning and to build RAG applications and agents. You can learn more about Amazon Bedrock here.

Prerequisite:

To integrate this, you need to have an AWS account. Setting up an AWS account is quite simple, so I won't cover that in this article. Let's get started.

In this article, we’ll explore two ways to integrate Bedrock into our application: using the SDK and the REST API provided by Bedrock. Before we start the integration, let's first obtain the Bedrock token.

AWS Bedrock Token:

Visit the Amazon Bedrock page and click on the Get started with Amazon Bedrock link. Log in with your existing account or create a new one. If you're not redirected after logging in, search for Bedrock on the AWS Console page and visit the service page.

In the left menu, select API keys under Discover (Discover→API keys). You'll find two options there: short-term and long-term keys. Generate one based on your needs.

Copy and save the API Key in a safe place, as you won't be able to copy it again unless you generate a new one.

Configuring Model:

Now that we have our API key, we need to set up the model we’re going to use.

Note: You will be charged based on the number of requests you make to each model. Each model has a different price for each API call. Check out the pricing for each model here.

In this implementation, we’ll use Amazon's own models, which are very low cost.

First, let’s see a simple text response with user input using Amazon’s Titan Text G1 Lite model. To configure a model, go to model access under configure and learn in the left catalog. Now you can see the different models available.

It’s important to note that these models are specific to the AWS region you’ve selected. You can see your AWS region in the top right corner. Not all models are available in all regions, and the AWS region is also important for integration.

To access any of the models, click on Available request. The availability can happen within seconds or sometimes takes 2–3 minutes. We’ll request the Titan Text G1 Lite for the us-east-1 (N. Virginia US) region.

You can also learn more about the model on the model catalog (Discover → model catalog) page.

After the model request is approved, go to the model catalog and search for the model we requested. On the detail page, you’ll see the model ID. Copy and save it, as it’s needed to access the model.

Now that we have the API key, AWS region, and model ID, let’s integrate Amazon Bedrock.

AWS SDK Method:

First, we'll implement the SDK method.

Install the package

npm i @aws-sdk/client-bedrock-runtime

Set the following values in your env file:

# AWS Configuration
AWS_REGION=
AWS_BEDROCK_MODEL=
AWS_BEARER_TOKEN_BEDROCK=

Copy the name exactly as it is, especially the AWS_BEARER_TOKEN_BEDROCK value, because the npm package will automatically read this value from an environment variable with that name, so we don't need to configure it manually.

Generating Text Response:

First, create a client with the AWS region using the BedrockRuntimeClient.

import {
  BedrockRuntimeClient
} from '@aws-sdk/client-bedrock-runtime';

const {
  AWS_REGION: awsRegion,
  AWS_BEDROCK_MODEL: modelId
} = process.env;

const client = new BedrockRuntimeClient({
  region: awsRegion
});

Then, using the ConverseCommand, we'll create a command that includes the user's message, the model ID of the model we want to use, and inference settings to control the outputs.

import {
  BedrockRuntimeClient,
  ConverseCommand,
} from '@aws-sdk/client-bedrock-runtime';

const inferenceConfig = {
  maxTokens: 4096,
  temperature: 0.5,
  topP: 0.9,
};

const conversation = [{
  role: 'user',
  content: [{
    text: 'Tell me about Max Verstappen.'
  }]
}, ];

const command = new ConverseCommand({
  modelId,
  messages: conversation,
  inferenceConfig,
});

As you can see, the user message is formatted by specifying the role and including the prompt in the content. The role is especially useful for multi-turn chat, which we’ll discuss later in the article.

Next, we’ll use the client we set up earlier to execute the command.

const response = await client.send(command);

For the input above, the Amazon Bedrock response will look something like the response below.

{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "f871265d-b765-4c8e-ba69-f0f008fc81df",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "metrics": {
    "latencyMs": 8472
  },
  "output": {
    "message": {
      "content": [
        {
          "text": "Max Verstappen is a Belgian-Dutch racing driver who currently competes in the Formula One World Championship for Red Bull Racing. He is the son of former Formula One driver Jos Verstappen.\n\nBorn on September 30, 1997, in Hasselt, Belgium, Max Verstappen began karting at a young age and quickly rose through the junior ranks. He won several championships, including the European KF1 Championship and the World KZ Championship.\n\nVerstappen made his Formula One debut in 2015 with Toro Rosso, becoming the youngest driver to compete in the sport at the age of 17 years and 166 days. He scored his first points in his second race and went on to finish the season with 49 points.\n\nIn 2016, Verstappen was promoted to Red Bull Racing, replacing Daniil Kvyat. He won his first Grand Prix at the 2016 Spanish Grand Prix, becoming the youngest driver to win a Formula One race at the age of 18 years and 228 days.\n\nSince then, Verstappen has become one of the most successful and dominant drivers in the sport. He has won numerous Grands Prix, including multiple victories at tracks like Monaco, Austria, and Brazil. He has also been a consistent championship contender, finishing in the top three of the drivers' championship in several seasons.\n\nIn 2021, Verstappen won his first Formula One World Drivers' Championship, defeating Lewis Hamilton in a thrilling and controversial season finale. He successfully defended his title in 2022, winning a record 15 Grands Prix and securing the championship with several races to spare.\n\nVerstappen is known for his aggressive driving style, which has sometimes led to controversy and criticism. However, he is also widely regarded as one of the most talented and skilled drivers in the sport, with a strong work ethic and a deep understanding of the technical aspects of racing.\n\nOff the track, Verstappen is known for his laid-back and down-to-earth personality. He is a popular figure among fans and has a strong social media presence. He is also involved in various charitable initiatives, including supporting the Wings for Life Foundation, which aims to find a cure for spinal cord injuries."
        }
      ],
      "role": "assistant"
    }
  },
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 43,
    "outputTokens": 452,
    "totalTokens": 495
  }
}

From the above response, we can extract the text part of the content array. Also, notice that the role is labeled as assistant (some providers use model like Gemini).

Multi-Turn Conversation:

Sometimes, just generating a response isn't enough, and we need a multi-turn conversation to handle follow-up prompts.

For a multi-turn chat, each time we use the Bedrock client, we must include the previous conversation with roles. The user role is for our prompt, and the assistant role is for the AI-generated response. The entire conversation is then sent in the message parameter.

You can find the implementation below.

const buildMessage = (role, text) => ({
  role,
  content: [{
    text
  }],
});

const inferenceConfig = {
  maxTokens: 4096,
  temperature: 0.5,
  topP: 0.9,
};

const firstUserMessage = 'Tell me about Max Verstappen.';
const firstConversation = [buildMessage('user', firstUserMessage)];

const firstCommand = new ConverseCommand({
  modelId,
  messages: firstConversation,
  inferenceConfig,
});

const firstResponse = await client.send(firstCommand);
const firstAssistantReply = firstResponse.output.message.content[0].text;
console.log(firstAssistantReply);

const secondUserMessage = 'What team does he drive for?';
const secondConversation = [
  buildMessage('user', firstUserMessage),
  buildMessage('assistant', firstAssistantReply),
  buildMessage('user', secondUserMessage),
];

const secondCommand = new ConverseCommand({
  modelId,
  messages: secondConversation,
  inferenceConfig,
});

const secondResponse = await client.send(secondCommand);

The output will look something like this:

{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "c7bf1210-095b-48ff-b450-1e02aa1160b0",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "metrics": {
    "latencyMs": 825
  },
  "output": {
    "message": {
      "content": [
        {
          "text": "Max Verstappen drives for the Red Bull Racing team in Formula One."
        }
      ],
      "role": "assistant"
    }
  },
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 471,
    "outputTokens": 16,
    "totalTokens": 487
  }
}

With multi-turn chat, each follow-up prompt will use more tokens, so keep an eye on this if you're using it.

Streaming Response:

When using the generated output response, for example on the front end, we usually have to wait for the full response. This isn't ideal for long responses.

To solve this, we can use response streaming, which is supported by Amazon Bedrock.

For streaming, we need to use the ConverseStreamCommand instance instead of ConverseCommand in the implementation shown below.

import {
  ConverseStreamCommand
} from '@aws-sdk/client-bedrock-runtime';

const conversation = [{
  role: 'user',
  content: [{
    text: 'Tell me about Max Verstappen, Red Bull Racing and his career over the years.'
  }, ],
}, ];

const command = new ConverseStreamCommand({
  modelId,
  messages: conversation,
  inferenceConfig,
});

const response = await client.send(command);

for await (const item of response.stream) {
  if (item.contentBlockDelta?.delta?.text) {
    const chunk = item.contentBlockDelta.delta.text;
    console.log(`${chunk}\n----`);
    yield chunk;
  }
}

Here, I've added logging to check each chunk coming from the streaming response. To see the streaming visually, render it on the front end or use cUrl in the terminal by adding this to an API route.

Structured Output:

In most cases, we need structured data for better clarity and consistency. This is especially useful when we plan to use the response to write our own logic.

Some Amazon Bedrock models support structured data output, and Bedrock uses tool use to get structured output. Although more functionality can be achieved with tool use, you can read more about it in the docs.

Since not all models support generating structured output, check which models support tool use in the documentation here before proceeding.

Implementation:

With tool use, we need to define a toolSpec for the structured data we want, then pass this tool to the Bedrock client to generate a response.

For example, let's look at the following toolSpec for a better understanding.

const driverDetailsTool = [{
    toolSpec: {
      name: 'race_driver_info',
      description: "'Returns information about a Formula 1 driver in structured format',"
      inputSchema: {
        json: {
          type: 'object',
          properties: {
            about: {
              type: 'string'
            },
            name: {
              type: 'string'
            },
            birthDate: {
              type: 'string'
            },
            nationality: {
              type: 'string'
            },
            team: {
              type: 'string'
            },
            championshipsWon: {
              type: 'integer'
            },
          },
          required: ['name', 'birthDate', 'nationality', 'team', 'championshipsWon'],
        },
      },
    },
  }, ],

  const courseTool = [{
    toolSpec: {
      name: 'javascript_course_generator',
      description: "'Generates a structured JavaScript course with multiple chapters',"
      inputSchema: {
        json: {
          type: 'object',
          properties: {
            courseTitle: {
              type: 'string',
              description: "'Title of the course',"
            },
            chapters: {
              type: 'array',
              description: "'List of course chapters',"
              items: {
                type: 'object',
                properties: {
                  title: "{"
                    type: 'string',
                    description: 'Title of the chapter'
                  },
                  description: "{"
                    type: 'string',
                    description: 'Description of the chapter'
                  },
                },
                required: ['title', 'description'],
              },
            },
          },
          required: ['courseTitle', 'chapters'],
        },
      },
    },
  }, ];

As seen above, we need to define the structured data we want under the toolSpec. Here are some key points to remember when creating a tool:

Provide a name and a clear description of the structured data we want.
Under the json key of the inputSchema object, we need to outline the structure.
For each field we want, include a type field for the data type and a description of what the data represents. Refer to the examples above for how to write concise descriptions.
If arrays or objects are needed, extend them with the properties key and follow the same type and description format, or extend further as shown above.
For each array or object type declared in the tool, include a required field for all the keys in that array or object to ensure the model recognizes all fields as required and responds without missing any keys (refer to the examples above).

After defining the tool, it needs to be sent with the bedrock runtime client we defined earlier. We’ll add the tool to the ConverseStreamCommand instance we created above, as shown in the following code:

const command = new ConverseStreamCommand({
  modelId,
  messages: conversation,
  inferenceConfig,
  toolConfig: {
    tools: courseTool // or use driverTool,
    toolChoice: { auto: {} },
  },
});

As you can see, we have added the toolConfig object and specified the tools and toolChoice.

You might wonder what toolChoice is, right? Well, tool use refers to using external tools that can help the model generate a response if needed. Setting it to auto allows the model to decide whether to use the external tool for generating the response or to proceed without it. Since we don't need any external tool, we can simply set toolChoice to auto.

For the two tool examples above, here’s the Bedrock response using the model meta.llama3-2-90b-instruct-v1:0.

Note: If you encounter the error "Bedrock API invocation error - on demand throughput isn't supported. Retry your request with the ID or ARN of an inference profile that contains this model." even if you provided the correct model ID, it may be due to “Cross-Region Inference”.

*If the model you used has a “*Cross-Region Inference” label next to it on the model availability request page in the AWS Bedrock console, you need to prefix it with the region like this:

For the model meta.llama3-2-90b-instruct-v1:0 for us, eu, or apac regions, you have to prefix the model ID with these, so it will look like this:

us regions: us.meta.llama3-2-90b-instruct-v1:0

eu regions: eu.meta.llama3-2-90b-instruct-v1:0

apac regions: apac.meta.llama3-2-90b-instruct-v1:0

I resolved this error using the solution from this blog. Check it out for more details.

Now, let’s see the two responses:

Using driverDetailsTool tool:

const driverPrompt = `
Extract structured information about the following Formula 1 driver.
Only return structured JSON using the provided schema.

Driver: Max Verstappen
`;

// Response
{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "06f666bb-eca4-48cb-bf4a-aa2fa579302e",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "metrics": {
    "latencyMs": 1370
  },
  "output": {
    "message": {
      "content": [
        {
          "toolUse": {
            "input": {
              "name": "Max Verstappen",
              "birthDate": "September 30, 1997",
              "nationality": "Dutch",
              "team": "Red Bull Racing",
              "championshipsWon": "2"
            },
            "name": "race_driver_info",
            "toolUseId": "tooluse_Ye7CjMrNQw6Da-MqO0QmRA"
          }
        }
      ],
      "role": "assistant"
    }
  },
  "stopReason": "tool_use",
  "usage": {
    "inputTokens": 288,
    "outputTokens": 59,
    "totalTokens": 347
  }
}

Using courseTooltool:

const coursePrompt = `
Generate a structured JavaScript course.
The course should have a title and a list of chapters.
Each chapter should include a title and a short description.

Only return structured JSON using the provided schema and return chapters as an array of JSON objects, not a stringified JSON.
`;

// Response
{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "026e8ab6-5688-440e-ba93-e13ee5cd232f",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "metrics": {
    "latencyMs": 1481
  },
  "output": {
    "message": {
      "content": [
        {
          "toolUse": {
            "input": {
              "courseTitle": "JavaScript Course",
              "chapters": [
                {
                  "title": "Chapter 1",
                  "description": "Introduction to JavaScript"
                },
                {
                  "title": "Chapter 2",
                  "description": "Data Types and Variables"
                }
              ]
            },
            "name": "javascript_course_generator",
            "toolUseId": "tooluse_cIS9FDwBTrCENcLKpxM8cA"
          }
        }
      ],
      "role": "assistant"
    }
  },
  "stopReason": "tool_use",
  "usage": {
    "inputTokens": 302,
    "outputTokens": 60,
    "totalTokens": 362
  }
}

Note: In the second example, I added the extra text “return chapters as an array of JSON objects, not a stringified JSON.” to the prompt. This is because if it's not specified, the chapters array might come as a stringified JSON object, which could cause issues since we would have to manually parse that data.

Now, with the help of the SDK, we have generated a normal text response, learned how to handle multi-turn chat, stream a response, and obtain structured data.

You can achieve the same result with the REST API endpoint if you prefer not to use the SDK. We'll explore this next.

REST API Method:

In the SDK Method above, I covered most of the terms and use cases, so here we won't go into detail about that. We'll just look at how to implement all four functionalities using the API endpoint through code.

Before moving forward, make sure you’ve added required environment variables in your env file.

# AWS Configuration
AWS_REGION=
AWS_BEDROCK_MODEL=
AWS_BEARER_TOKEN_BEDROCK=

Configuring API Endpoint:

The REST API endpoint is created dynamically using the AWS_REGION and AWS_BEDROCK_MODEL ID. Here's how the API URL is constructed:

const { AWS_REGION: region, AWS_BEDROCK_MODEL: modelId } = process.env;

const url = `https://bedrock-runtime.${region}.amazonaws.com/model/${modelId}/converse`;

Since I use the us-east-1 region and the amazon.titan-text-lite-v1 model (ID), here’s what the constructed API endpoint will look like:

https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.titan-text-lite-v1/converse

Text Response:

Using the above API endpoint, we can make a POST request to this endpoint using the AWS_BEARER_TOKEN_BEDROCK as a Bearer Authorization token in the headers. In the payload, we need to send the messages field, which is an array of objects. Each object should have a role type (user) and content (containing the user's prompt).

Here's how this implementation looks (I use axios, but you can also use the built-in fetch):

const {
  AWS_REGION: region,
  AWS_BEDROCK_MODEL: modelId,
  AWS_BEARER_TOKEN_BEDROCK: apiKey,
} = process.env;

const url = `https://bedrock-runtime.${region}.amazonaws.com/model/${modelId}/converse`;
const prompt = 'Tell me about Max Verstappen';
const payload = {
  messages: [
    {
      role: 'user',
      content: [{ text: prompt }],
    },
  ],
};

try {
  const response = await axios.post(url, payload, {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${apiKey}`,
    },
  });

  console.log(response);
  const content = response.data?.output?.message?.content;
  const text = content?.find((c) => c.text)?.text;
  console.log(text);
} catch (err) {
  console.error(err);
}

For the implementation above, this is what the output will look like:

{
  "metrics": {
    "latencyMs": 8268
  },
  "output": {
    "message": {
      "content": [
        {
          "text": "Max Verstappen is a Dutch Formula One racing driver who currently drives for the Red Bull Racing team. He was born on September 30, 1997, in Hasselt, Belgium, to former Formula One driver Jos Verstappen and his Belgian wife, Sophie Kumpen.\n\nVerstappen began karting at a young age and quickly rose through the junior ranks, winning several championships, including the European KF1 Championship and the World KF1 Championship. In 2014, he made his Formula One debut at the Australian Grand Prix, driving for the Toro Rosso team, becoming the youngest driver to compete in a Formula One World Championship at the age of 17 years and 166 days.\n\nIn 2016, Verstappen joined Red Bull Racing, replacing Daniil Kvyat, and has since become one of the most successful drivers in the sport. He has won numerous Grands Prix and has been a consistent championship contender.\n\nVerstappen's driving style is known for being aggressive and bold, often pushing the limits of his car and taking risks to gain an advantage. He has been involved in several high-profile incidents and controversies throughout his career, including a notable collision with Lewis Hamilton at the 2021 British Grand Prix.\n\nDespite the controversies, Verstappen has established himself as one of the top drivers in Formula One, known for his exceptional speed, skill, and determination. He won his first Formula One World Drivers' Championship in 2021, narrowly beating Lewis Hamilton in a dramatic and contentious season finale.\n\nVerstappen's success has been recognized with numerous awards and accolades, including the Lorenzo Bandini Trophy, the FIA Personality of the Year award, and the Dutch Sportsman of the Year award.\n\nOff the track, Verstappen is known for his laid-back and down-to-earth personality. He is an avid gamer and has participated in several esports events, including the F1 Esports Series. He is also involved in various charity initiatives, including supporting the Wings for Life Foundation, which aims to find a cure for spinal cord injuries.\n\nOverall, Max Verstappen is a talented and accomplished Formula One driver who has quickly become a fan favorite and a dominant force in the sport."
        }
      ],
      "role": "assistant"
    }
  },
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 42,
    "outputTokens": 450,
    "totalTokens": 492
  }
}

Multi Turn Chat:

For multi-turn chat, the messages array will contain the chat history. For each API call, you need to send the complete message history. This way, the model can understand the context of the previous chats and generate a response that fits the follow-up prompt.

In the implementation above, the message array looked like this:

const messages = [
  {
    role: 'user',
    content: [{ text: 'Tell me about Max Verstappen' }],
  },
];

Let's say the model responded to the above prompt with: “Max Verstappen is a Formula 1 race driver and is the current holder of the WDC (World Driver Championship).” If we want to add another follow-up prompt like “What team does he drive for?” then the messages array we need to send in the next API call will look like this:

const messages = [
  {
    role: 'user',
    content: [{ text: 'Tell me about Max Verstappen.' }],
  },
  {
    role: 'assistant',
    content: [
      {
        text: 'Max Verstappen is a Formula 1 race driver and is the current holder of WDC (World driver championship).',
      },
    ],
  },
  {
    role: 'user',
    content: [{ text: 'What team does he drive for?' }],
  },
];

Note: Use the role 'assistant' for the model's response.

Here's how we'll implement this in code:

const buildMessage = (role, text) => ({
  role,
  content: [{ text }],
});

const firstUserMessage = 'Tell me about Max Verstappen.';
const firstConversation = [buildMessage('user', firstUserMessage)];

const firstResponse = await axios.post(
  url,
  { messages: firstConversation },
  {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${apiKey}`,
    },
  }
);

const firstAssistantReply = firstResponse.data.output.message.content?.find((c) => c.text)?.text;

console.log(firstAssistantReply);

const secondUserMessage = 'What team does he drive for?';
const secondConversation = [
  buildMessage('user', firstUserMessage),
  buildMessage('assistant', firstAssistantReply),
  buildMessage('user', secondUserMessage),
];

const secondResponse = await axios.post(
  url,
  { messages: secondConversation },
  {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${apiKey}`,
    },
  }
);

const secondAssistantReply = secondResponse.data.output.message.content?.find((c) => c.text)?.text;

console.log(secondAssistantReply);

Above, I added the buildMessage function to make it easier to create message objects for the chat history in the messages array.

Here’s what the output looks like for the implementation above:

// First Response:

{
  "metrics": {
    "latencyMs": 6946
  },
  "output": {
    "message": {
      "content": [
        {
          "text": "Max Verstappen is a Dutch Formula One racing driver who currently drives for the Red Bull\nRacing team. He was born on September 30, 1997, in Hasselt, Belgium. Verstappen began\nkarting at a young age and quickly rose through the junior ranks, winning several\nchampionships.\n\nIn 2014, Verstappen made his Formula One debut with the Toro Rosso team at the age of 17,\nbecoming the youngest driver to compete in the sport at the time. He scored his first points\nin his second Grand Prix and went on to finish the season with 49 points.\n\nIn 2016, Verstappen moved to Red Bull Racing, replacing Daniil Kvyat. He won his first Grand\nPrix at the 2016 Spanish Grand Prix, becoming the youngest driver to win a Formula One\nWorld Championship Grand Prix at the time.\n\nVerstappen has since become one of the most successful drivers in the sport, winning\nnumerous Grands Prix and finishing as the runner-up in the 2020 and 2021 World Drivers'\nChampionships. In 2021, he won his first World Drivers' Championship, defeating Lewis\nHamilton in a dramatic and contentious season finale.\n\nVerstappen is known for his aggressive driving style and his ability to push his car to the\nlimit. He has also been involved in several high-profile controversies throughout his career,\nincluding a number of on-track incidents and disputes with other drivers.\n\nDespite these controversies, Verstappen remains one of the most popular and respected\ndrivers in the sport, with a large and dedicated fan base. He is also known for his close\nrelationship with his father, Jos Verstappen, who is a former Formula One driver and has\nplayed a significant role in his son's career.\n\nIn 2022, Verstappen successfully defended his World Drivers' Championship title, winning 15\nGrands Prix and finishing 146 points clear of his nearest rival, Charles Leclerc."
        }
      ],
      "role": "assistant"
    }
  },
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 43,
    "outputTokens": 393,
    "totalTokens": 436
  }
}

// Second response

{
  "metrics": {
    "latencyMs": 658
  },
  "output": {
    "message": {
      "content": [
        {
          "text": "Max Verstappen drives for the Red Bull Racing team in the Formula One World\nChampionship."
        }
      ],
      "role": "assistant"
    }
  },
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 452,
    "outputTokens": 19,
    "totalTokens": 471
  }
}

Streaming Response:

To use the response as a stream, we need to use a different endpoint, although the dynamic construction of the API URL remains the same. The endpoint for streaming needs to be changed to converse-stream instead of converse at the end. The new API URL constructed for streaming will look like the following:

https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.titan-text-lite-v1/converse-stream

To learn more about the converse-stream endpoint, you can check out this documentation.

Since axios doesn’t support fetching stream data, we’ll use the built-in fetch method. Here is what the implementation will look like:

const {
  AWS_REGION: region,
  AWS_BEDROCK_MODEL: modelId,
  AWS_BEARER_TOKEN_BEDROCK: apiKey,
} = process.env;

const url = `https://bedrock-runtime.${region}.amazonaws.com/model/${modelId}/converse-stream`;
const streamConverse = async (prompt) => {
  const fetch = global.fetch;

  const payload = {
    messages: [
      {
        role: 'user',
        content: [{ text: prompt }],
      },
    ],
  };

  try {
    const response = await fetch(url, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${apiKey}`,
        Accept: 'text/event-stream',
      },
      body: JSON.stringify(payload),
    });

    if (!response.ok || !response.body) {
      throw new Error(`HTTP ${response.status} - ${response.statusText}`);
    }

    const reader = response.body.getReader();
    const decoder = new TextDecoder('utf-8');

    console.log('[Stream Start]');
    let buffer = '';
    const allChunks = [];

    while (true) {
      const { done, value } = await reader.read();
      if (done) {
        console.log('[Stream End]');
        console.log('\n[Final Response]:', allChunks.join(''));
        console.log('\n[Total Chunks]:', allChunks.length);
        break;
      }

      const chunk = decoder.decode(value, { stream: true });
      buffer += chunk;

      const lines = buffer.split('\n');
      buffer = lines.pop() || '';

      for (const line of lines) {
        const cleanLine = line.replace(/[^\x20-\x7E\n\r\t]/g, '').trim();

        if (!cleanLine) continue;

        let searchIndex = 0;
        while (true) {
          const messageTypeIndex = cleanLine.indexOf(':message-typeevent', searchIndex);
          if (messageTypeIndex === -1) break;

          const jsonStart = messageTypeIndex + ':message-typeevent'.length;

          let braceCount = 0;
          let jsonEnd = jsonStart;
          let foundStart = false;

          for (let i = jsonStart; i < cleanLine.length; i++) {
            const char = cleanLine[i];
            if (char === '{') {
              if (!foundStart) foundStart = true;
              braceCount++;
            } else if (char === '}') {
              braceCount--;
              if (foundStart && braceCount === 0) {
                jsonEnd = i + 1;
                break;
              }
            }
          }

          if (foundStart && braceCount === 0) {
            const jsonString = cleanLine.substring(jsonStart, jsonEnd);

            try {
              const data = JSON.parse(jsonString);

              if (data?.delta?.text) {
                console.log('[Chunk]:', data.delta.text);
                allChunks.push(data.delta.text);
              }
            } catch (err) {
              console.error('[JSON Parse Error]:', err.message, 'Raw JSON:', jsonString);
            }
          }

          searchIndex = messageTypeIndex + 1;
        }
      }
    }
  } catch (err) {
    console.error('[Stream Error]', err.message);
  }
};

Testing a streaming implementation can be difficult, so I recommend using this implementation in an API and using a terminal to log each chunk of the response coming from the stream.

Structured Output:

To retrieve structured data output, we need to include a toolConfig (also known as tool use) field in the payload, which contains the tool—a JSON schema of how we want the structured output to be.

For more information about tool use, please check the Structure Output section in the SDK implementation above. To implement this, we must define a tool (a JSON schema of the output format) and include it in the payload. Here is how the implementation would look:

const prompt = `Generate a structured JavaScript course.
The course should have a title and a list of chapters.
Each chapter should include a title and a short description.

Only return structured JSON using the provided schema and return chapters as an array of JSON objects, not a stringified JSON.`;

const messages = [
  {
    role: 'user',
    content: [{ text: prompt }],
  },
];

const tools = [
  {
    toolSpec: {
      name: 'javascript_course_generator',
      description: "'Generates a structured JavaScript course with multiple chapters',"
      inputSchema: {
        json: {
          type: 'object',
          properties: {
            courseTitle: {
              type: 'string',
              description: "'Title of the course',"
            },
            chapters: {
              type: 'array',
              description: "'List of course chapters',"
              items: {
                type: 'object',
                properties: {
                  title: "{ type: 'string', description: "'Title of the chapter' },\""
                  description: "{ type: 'string', description: 'Description of the chapter' },"
                },
                required: ['title', 'description'],
              },
            },
          },
          required: ['courseTitle', 'chapters'],
        },
      },
    },
  },
];

const payload = {
  messages,
  toolConfig: {
    tools,
    toolChoice: { auto: {} },
  },
};

try {
  const response = await axios.post(url, payload, {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${apiKey}`,
    },
  });

  const content = response.data?.output?.message?.content || [];
  console.log(content);
} catch (err) {
  console.error(err);
}

For this implementation, the result would look like the following:

{
  "metrics": {
    "latencyMs": 2630
  },
  "output": {
    "message": {
      "content": [
        {
          "toolUse": {
            "input": {
              "courseTitle": "JavaScript Fundamentals",
              "chapters": [
                {
                  "title": "Chapter 1: Introduction to JavaScript",
                  "description": "This chapter introduces the basics of JavaScript, including variables, data types, and functions."
                },
                {
                  "title": "Chapter 2: Control Structures",
                  "description": "This chapter covers control structures in JavaScript, such as conditional statements and loops."
                },
                {
                  "title": "Chapter 3: Functions",
                  "description": "This chapter explores functions in JavaScript, including function declarations, function expressions, and arrow functions."
                }
              ]
            },
            "name": "javascript_course_generator",
            "toolUseId": "tooluse_RdFMcX8GTMGUqxVNFsXubQ"
          }
        }
      ],
      "role": "assistant"
    }
  },
  "stopReason": "tool_use",
  "usage": {
    "inputTokens": 342,
    "outputTokens": 125,
    "totalTokens": 467
  }
}

Note: The data here may not be accurate because the model's training data is outdated.

Now that we have implemented text generation, multi-turn chat, streaming responses, and retrieving structured data using the REST API method, you can proceed to implement Amazon Bedrock using any of these SDKs or REST API methods.

You can also find this entire implementation in my aws-bedrock GitHub repo, where I’ve included both the SDK and REST API versions.

I hope this article helps you learn how to implement Amazon Bedrock in your JavaScript project. We've explored two methods: using the SDK and the REST API to generate responses. I hope you found this helpful. I would love to hear your feedback, so if you encounter any issues while implementing or notice any mistakes, please feel free to comment or contact me. Thank you. Happy coding! :)

Understanding SAML Authentication: Key Concepts and Differences Between SP and IdP Initiated Flows

Dushmanta — Fri, 07 Mar 2025 12:58:07 +0000

Most enterprises work with different services like Slack, Microsoft 365, Salesforce, and Google Workspace daily. So, have you ever wondered how these enterprises provide access to all these services in one dashboard for their employees? Or imagine you have an enterprise and want to give your employees access to these platforms.

Since these services are independent, they would typically require separate login credentials. However, managing different usernames and passwords for each service would be an overhead and not an efficient solution. Can you imagine handling 10 different login credentials if there are 10 different services? This not only creates unnecessary complexity but also becomes a bottleneck in productivity and security management.

This is where SAML comes into the picture. SAML authentication solves this overhead by enabling seamless access to multiple services with a single set of credentials (the credentials of their own organization / identity provider). In this article, let’s learn about SAML, how it works, and the important terms related to SAML authentication that you need to know.

What is SAML?

SAML, or Security Assertion Markup Language, is an open standard based on XML (Extensible Markup Language) used for authentication and sharing identity data. SAML transmits authentication and identity data using an XML data structure over HTTP or SOAP methods. This data is exchanged between an organization’s Identity Provider (IdP) and a Service Provider (SP)—which we will discuss next.

Identity Provider (IdP):

In SAML authentication, the Identity Provider (IdP) is the entity that provides identity information about the user who needs authentication. The IdP can be your organization itself, or if you're using a third-party service like Google or Okta, then it will act as the IdP.

Service Provider (SP):

The Service Provider (SP) is the entity that offers services to the organization and its users. For example, Microsoft 365 and Google Workspace act as Service Providers for organizations integrating with them. Any platform that provides a service and support SAML authentication can be considered an SP.

For example, imagine you have an organization let’s say Acme Inc., and you want to offer health benefits to employees through a health benefits provider let’s say Healthy Inc. Your employees need to log in to Healthy Inc.'s portal to access their benefits. If you establish SAML authentication, Acme Inc. acts as the IdP, providing identity data for employees, while Healthy Inc. acts as the SP, granting access based on that authentication.

How SAML Works?

Establishing Trust Between IdP and SP

SAML authentication occurs between an Identity Provider (IdP) and a Service Provider (SP). Before the authentication process begins, both entities must establish a trust relationship. This is achieved by exchanging metadata, specifying endpoints, defining connection methods, and sharing signing and encryption certificates based on Public Key Infrastructure (PKI).

Let’s explore some key terms required for trust establishment.

ACS URL (Assertion Consumer Service URL):

The ACS URL is provided by the Service Provider (SP) and serves as the endpoint where users are redirected to complete authentication and access the service. Typically, SAML authentication requests to this endpoint are made via a POST request.

IdP Single-Sign On (SSO) / Login URL:

The IdP SSO/Login URL is provided by the Identity Provider (IdP) to the Service Provider (SP) and serves as the endpoint that receives the SAML authentication request from the SP. This URL is where users are redirected when they attempt to log in via SAML authentication to the SP using ACS URL, allowing the SP to obtain authentication data from the IdP.

Audience URI / Entity ID:

The Audience URI is provided by the SP to the IdP and serves as a unique identifier representing the SP. This URI indicates who the assertion is intended for, and it is validated by the Service Provider. Typically, the Audience URI is included in the SAML response data.

Relay State:

The Relay State is a parameter provided by the SP to the IdP. It is a URL or identifier used to maintain the user's state. The Relay State determines where to redirect the user after successful SAML authentication. Since the SP doesn't know the user's state before the SAML authentication login, the Relay State holds a value that guides where the user is redirected after successful authentication. In most cases, the Relay State is a URL where the user will be redirected.

Public-Private Key Pair:

A public-private key pair (Public Key Infrastructure) is used to secure the SAML authentication process through encryption and signature.

Both the IdP and SP generate their own public-private key pairs and share their public keys with each other.
These keys are used to sign the requests and responses with private keys, and they can be verified using the shared public keys. This ensures that the requests or responses are legitimate and have not been tampered with.

These above configuration are used to establish trust between the IdP and SP.

SAML Authentication Flow

Next, let’s explore how SAML authentication works.

User Request Access

The user tries to access a service (SP) like Healthy Inc.
SP Redirects User to IdP SSO URL

The SP generates a SAML Authentication Request and redirects the user to the IdP SSO Login URL.
User Authenticates with IdP

The user logs into the Identity Provider (IdP) (Acme Inc. in this example).
IdP Generates SAML Response

Upon successful authentication, the IdP creates a SAML Response, including a SAML Assertion (more about SAML Response and Assertion are discussed below) containing the user’s identity data. Then the response is digitally signed with the IdP’s private key to ensure authenticity.
IdP Sends Response to SP

The IdP redirects the user back to the SP’s ACS (Assertion Consumer Service) URL with the SAML Response, including the Relay State, via an HTTP POST request.
SP Validates the SAML Response

The SP:
- Verifies the digital signature using the IdP’s public key (which was provided during trust establishment).
- Ensures that the response is valid and hasn’t been tampered with.
Redirects to Relay State

After successful verification, the SP redirects the user to the Relay State URL (provided in the initial request). This ensures the user lands on the correct page after authentication.

Here’s an image illustrating SAML authentication flow:

SAML Response and Assertion

What is a SAML Response?

A SAML Response is an XML-based authentication message sent by the Identity Provider (IdP) to the Service Provider (SP). It contains the SAML Assertion, which holds user identity data, attributes, and other important information such as the signature, issuer, destination, and timestamp.

What is a SAML Assertion?

A SAML Assertion is the core part of the SAML Response, as it carries the user’s identity information to the SP. The SP uses this assertion to create a login session and grant access to the requested service.

SAML Signature

Before sending the SAML Response, the IdP signs it using its private key. The response includes:

Digest value (ensuring data integrity)
Signature value (confirming authenticity)
X.509 certificate (used for validation)

Additionally, the SAML Assertion itself can also be signed with the same key. This means:

Both the SAML Response and the SAML Assertion can be signed separately.
The digest value, signature, and X.509 certificate are included in both if needed.

For better security, it is recommended to sign both the SAML Assertion and the SAML Response to prevent tampering.

Structure of SAML Response

A typical SAML response contains the following data:

Issuer → Identifies which IdP issued the SAML Response.
Status → Indicates whether the authentication succeeded or failed.
Assertions → Contains authentication, attribute, and assertion authorization details.
Signature & Encryption → Contains the signature data for the whole SAML response.

SP-Initiated vs. IdP-Initiated Flow

Now that we understand how SAML works, it's important to know that there are two types of authentication flows in which SAML can be implemented:

SP-Initiated Flow → The user starts at the Service Provider (SP).
IdP-Initiated Flow → The user starts at the Identity Provider (IdP).

SP-Initiated Flow:

The SP-Initiated Flow is the most commonly used SAML authentication flow. The SAML authentication process we discussed earlier follows this flow, where the user starts from the Service Provider's login page.

How it works:

The user tries to log in from the SP (e.g., Healthy login page).
The SP redirects the user to the IdP’s SSO Login URL, sending a SAML Authentication Request.
The IdP authenticates the user.
The IdP generates a SAML Response and sends it back to the SP (via ACS URL).
The SP validates the response and logs the user in.

Note: In an SP-Initiated Flow, the authentication request starts from the SP, and the SAML Authentication Request is sent from the SP to the IdP.

Here’s an image illustrating the SP-Initiated SAML authentication flow:

IdP-Initiated Flow:

The IdP-Initiated Flow is less common but is often used in enterprise dashboards, where employees log in once and access multiple services from a single portal (Single Sign-On).

The user logs in directly at the IdP (e.g., a corporate dashboard).
When the user wants to access a specific service, the IdP generates a SAML Response and sends it directly to the SP (without a request).
The SP validates the response and logs the user in.

Note: In an IdP-Initiated Flow, there is no authentication request sent from the SP. The IdP directly initiates the authentication process and sends the SAML Response to the SP.

Here’s an image illustrating the IdP-Initiated SAML authentication flow:

Benefits of SAML Auth

Here are some key benefits of using SAML authentication:

Eliminates the need for multiple passwords, reducing the risk of phishing and credential theft.
Users only need to log in once to access multiple service providers, reducing password fatigue.
Users don’t have to remember multiple credentials, leading to faster and smoother access.
Enterprises can use their own IdP to manage authentication securely across various platforms.
Service providers don’t need to store or manage user credentials in their database, reducing security risks and compliance overhead.

How SAML is Different from Oauth?

Now that we've learned about SAML, you might wonder how it differs from OAuth and whether OAuth can be used for login to multiple service providers instead.

Both SAML and OAuth are open standards, but they serve different purposes:

SAML is primarily used for authentication, verifying a user’s identity.
OAuth is used for authorization, granting access to resources without exposing login credentials.

Example: Let’s say you want to log in to multiple services—Microsoft 365, Slack, and Salesforce—using OAuth with Google instead of SAML. Here’s what happens:

You visit Microsoft 365 and choose to log in with Google.
Google redirects you to an OAuth consent page asking for permission.
Once you grant permission, Microsoft 365 receives an OAuth access token and logs you in.

Now, if you want to log in to more service providers, you have to repeat the same process for each provider separately. Not to mention, your IdP must be Google since you’re using Google OAuth. This becomes difficult for enterprises that have their own Identity Provider (IdP).

Now, if we use SAML instead,

We can integrate multiple service providers with a single enterprise IdP.
Users can log in once and seamlessly access all service providers from one dashboard through SAML authentication.

Conclusion

As we learned above, SAML authentication is a secure way to access multiple independent services. It helps enterprises provide access to multiple services from their IdP. It enables seamless SSO, reduces security risks, and simplifies identity management. It’s also worth mentioning that SAML is not limited to enterprises—it can be used by any organization or system that requires secure and efficient authentication across multiple service providers.

I hope this article helped you understand the key concepts of SAML, its different authentication flows, and its benefits. Please feel free to comment if you have any doubts or reach out to me. Also, let me know if I’ve made any mistakes in the article—I’d love to learn and improve! Happy coding :)

Step-by-Step Guide to Hosting Your Website on a VPS Using Caddy Server and Cloudflare

Dushmanta — Mon, 11 Nov 2024 23:14:10 +0000

Hosting your frontend application on Vercel or Netlify is quite simple; you just need to provide access to your repository and the build command. Recently, I moved my personal portfolio from Next.js to Astro because my portfolio has more static content. My previous Next.js site was deployed on both Vercel and Netlify, so it was easy to deploy by just changing the build command. However, since I recently bought a Virtual Private Server (VPS) from Digital Ocean (Droplet), I decided to host my portfolio there and serve it through Cloudflare.

In this article, we'll explore how I achieved this using my VPS, Caddy server, and Cloudflare, and how you can do it too by following the steps. But first, let's briefly understand what the Caddy server and Cloudflare are.

Cloudflare:

You might already know about Cloudflare, a well-known company that offers services to enhance website performance and security. I'm using Cloudflare's reverse proxy service, which is nearly free. You might wonder, what is a reverse proxy? Simply put, a reverse proxy is a server that sits between my VPS and the user. It acts as a protective layer between my website and the public, helping to hide my server's IP address and prevent certain types of malware and cyberattacks.

Caddy server:

Caddy server is a free and open-source web server written in Golang. You might have heard of Apache or Nginx web servers, right? Caddy is similar to those, but it is secure by default. Caddy automatically gets and renews TLS certificates, making secure HTTPS connections easy.

P.S. I should mention that I already have my domain and a Cloudflare account. I also purchased a VPS from DigitalOcean with a Debian setup. This article won't cover these setups.

Adding VPS IP to Cloudflare

First, go to your Cloudflare dashboard and, in the Website section, add the domain name you want to host on your VPS. I've already added mine, and it should look something like this:

First, let's set up our SSL/TLS encryption. On the Cloudflare dashboard, click on the domain name you added (for example, dushmanta.dev in my case). This will open the domain overview page and settings, which should look like this:

Set SSL/TLS encryption mode:

On the domain settings page, go to the SSL/TLS section (refer to the image above). It should look like this:

From the SSL/TLS overview page, go to the configuration settings. It should look something like this:

In this section, set the custom SSL/TLS to "Full (Strict)" and save it.

The reason to choose "Full (Strict)" is to ensure the request connection is encrypted from end to end. This means when someone visits your domain, the request goes to Cloudflare, and then Cloudflare serves the content by connecting to your VPS. The flow looks like this:

User → Cloudflare: HTTPS

Cloudflare → Your VPS: HTTPS with a validated certificate

VPS → Cloudflare

Cloudflare → User (HTTPS)

Get Your Droplet (VPS) Ipv4

Now, let's get the IP of the VPS from the DigitalOcean dashboard. The dashboard should look like this, and you need to copy your IPv4 address.

Add “A” records to Cloudflare DNS

On the domain settings page, go to the DNS section.

Now, if you have used Cloudflare for your website before, you might have some "A", "AAAA", "CNAME", and "TXT" records. In my case, since I deployed my site with Netlify, my DNS records from Netlify looked something like this:

Before adding our VPS server's IP, we need to delete all the "A", "AAAA", and "CNAME" records (only for the root domain) if they exist. After removing them, we need to add "A" and "CNAME" DNS records pointing to the VPS IP address.

"A" record: Name → @, VPS IPv4 address, Proxy status → on
"CNAME" record: Name → WWW, Target → Original domain name, Proxy status → on

When adding a new DNS record, set the type to "A". In the Name section, add @, which represents your website (in my case, it’s dushmanta.dev), and ensure the proxy status is turned on. Then save it. It should look like this.

Next, add a "CNAME" record named "WWW" to create an alias. This way, if someone accesses the website using www.dushmanta.dev, it will redirect to the main website. Set your website as the target and ensure the proxy status is turned on. It should look like this.

Now that our Cloudflare DNS is set up, let's move on to build the portfolio from the repository and configuring Caddy on our VPS to manage the requests.

Build the Portfolio on VPS

Create a clone of your repository from GitHub, GitLab, or wherever your portfolio code is hosted onto your VPS, if you haven’t already. Next, run a build of your portfolio using the required command to generate the output in the dist folder, if you’re using a framework. If you're using vanilla JavaScript with HTML and CSS, simply place your HTML, CSS, and JS files in the dist folder. Now, the build output will be in the dist folder.

Now we can serve the build files from this repo, but for better organization, I recommend creating a separate folder for the build files. This way, if we want to host more websites on our VPS in the future, it will be easier to manage.

Create a Directory

On your VPS, create a directory for your site with the following command. (Since I’m hosting my portfolio, I will name the directory "portfolio.")

sudo mkdir -p /var/www/your-site
# sudo mkdir -p /var/www/portfolio

Copy dist Folder

Next, copy the build files from the repository to the dedicated directory using the following command:

sudo cp -r dist/* /var/www/your-site/
# sudo cp -r dist/* /var/www/portfolio/

Install and Set Up Caddy Server

Install the Caddy server on your VPS according to your distribution by following the installation docs. There are two releases: a stable release and a testing release. It is recommended to use the stable release to avoid any issues. My server is Debian, so I will install the apt package.

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

Set Proper Permission

Next, give Caddy the correct permissions for the folder where the build files are located using the following command:

sudo chown -R caddy:caddy /var/www/your-site
# sudo chown -R caddy:caddy /var/www/portfolio

Setting Up Caddy Service Configuration

Now, let's verify or create the systemd Caddy service configuration. This setup ensures that the systemd service file automatically manages Caddy's lifecycle, which includes automatic starts, restarts, and proper resource management on the server.

Open the service file with the following command (this command will create the file if it doesn't exist):

sudo nano /etc/systemd/system/caddy.service

The service file should look something like this:

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/caddy run --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target

If it's empty, update the file with the contents provided above and be sure to save it.

Notes:

This configuration makes sure Caddy starts automatically with your system.
It waits for network connectivity before starting.
The service is set up to handle high loads with LimitNOFILE.

After saving the configuration, make sure to follow the next steps:

Reload systemd daemon:

sudo systemctl daemon-reload

Start Caddy service:

sudo systemctl start caddy

Check status:

sudo systemctl status caddy

Enable auto-start:

sudo systemctl enable caddy

Configure the Caddyfile

After setting the correct permissions and configuring the systemd Caddy service file, let's set up the Caddyfile. This is where we configure our domain settings and file serving options. To learn more about the Caddyfile, check out the official documentation.

To edit the Caddyfile, run the following command:

sudo nano /etc/caddy/Caddyfile

Basic Configuration Structure:

Domain names (both root and www)
TLS configuration for Cloudflare SSL
Root directory path (your build folder)
File server directive (helps serve web pages from the directory mentioned above)

Here's how the Caddyfile should look:

example.com, www.example.com {
    # TLS configuration for Cloudflare
    tls {
        protocols tls1.2 tls1.3
    }

    root * /var/www/site-name # the build folder
    file_server # File server directive
}

Example Implementation:

For my website, the Caddyfile looks like this:

dushmanta.dev, www.dushmanta.dev {
    tls {
        protocols tls1.2 tls1.3
    }
    root * /var/www/portfolio
    file_server
}

Save the configuration file and then restart the Caddy service with the following command:

sudo systemctl restart caddy

And voilà! You've set up your Caddy server on your VPS, and it's ready to be served through Cloudflare.

Testing Your Configuration

Now, test if it works by using curl for both DNS records.

curl -v https://example.com
curl -v https://www.example.com
# Example
# curl -v https://dushmanta.dev
# curl -v https://www.dushmanta.dev

If you find any errors, make sure to configure your Caddy service and Caddyfile correctly, and check that your DNS settings are correct in your Cloudflare dashboard.

I hope this article helps if you're trying to host your website on your personal VPS with Caddy server and Cloudflare. Please let me know if you find any mistakes or have suggestions for improvement. Thank you, and happy coding! :)

DEV Community: Dushmanta

Node.js Multi-Threading with Worker Threads

SIngle-threaded JavaScript:

Single vs. Multi-threading:

Example:

How Multi-Threading Works on Single-Core vs Multi-Core CPUs:

Single-core:

Multi-core:

Concurrency vs. Parallelism:

When to Use Multi-Threading in Node.js

Introducing the Node.js Worker Threads Module

Process vs. Thread:

Worker Threads:

Implementing Worker Thread Module

Note:

Transform PostgreSQL into a Vector Database with pgvector

Prerequisite:

Installation:

Find PostgreSQL Container

Installing pgvector

Enable pgvector

Testing

Create Table

Insert Vector Embedding

Vector Search

Delete Table

Amazon Bedrock with JavaScript: Text, Chat, Streaming & Structured Output via SDK and REST

What is Amazon Bedrock?

Prerequisite:

AWS Bedrock Token:

Configuring Model:

AWS SDK Method:

Generating Text Response:

Multi-Turn Conversation:

Streaming Response:

Structured Output:

Implementation:

REST API Method:

Configuring API Endpoint:

Text Response:

Multi Turn Chat:

Streaming Response:

Structured Output:

Understanding SAML Authentication: Key Concepts and Differences Between SP and IdP Initiated Flows

What is SAML?

Identity Provider (IdP):

Service Provider (SP):

How SAML Works?

Establishing Trust Between IdP and SP

ACS URL (Assertion Consumer Service URL):

IdP Single-Sign On (SSO) / Login URL:

Audience URI / Entity ID:

Relay State:

Public-Private Key Pair:

SAML Authentication Flow

SAML Response and Assertion

What is a SAML Response?

What is a SAML Assertion?

SAML Signature

Structure of SAML Response

SP-Initiated vs. IdP-Initiated Flow

SP-Initiated Flow:

IdP-Initiated Flow:

Benefits of SAML Auth

How SAML is Different from Oauth?

Conclusion

Step-by-Step Guide to Hosting Your Website on a VPS Using Caddy Server and Cloudflare

Cloudflare:

Caddy server:

Adding VPS IP to Cloudflare

Set SSL/TLS encryption mode:

Get Your Droplet (VPS) Ipv4

Add “A” records to Cloudflare DNS

Build the Portfolio on VPS

Create a Directory

Copy dist Folder

Install and Set Up Caddy Server

Set Proper Permission

Setting Up Caddy Service Configuration

Notes:

Installing `pgvector`

Enable `pgvector`