DEV Community: Yuval Hazaz

Modernizing Legacy Systems with Amplication's DB Schema Import

Yuval Hazaz — Sun, 20 Aug 2023 14:53:59 +0000

Modernizing legacy systems isn’t just about catching up with the latest and greatest tech; it’s about staying competitive. Yet, transitioning away from legacy systems can be daunting, especially with the risk of losing valuable data or facing compatibility issues. The effort involved in setting up numerous data models, crafting Data Transfer Objects (DTOs), and coding boilerplate for hundreds of existing tables, not to mention managing intricate relations, can be massive and time-consuming. This is where our latest feature - DB Schema Import - steps in, offering an efficient and seamless way to bring your old infrastructure into the modern era without getting lost in the complexities.

Why modernize with Amplication?

Rapid Development

Launch your projects faster. Amplication’s auto-generation tools drastically cut down the development process by handling boilerplate and infrastructure code, saving you months that you'd otherwise spend crafting code for your existing data models.

Consistency

No more patchwork solutions. Amplication ensures your backend services are consistent, scalable, and up to industry standards.

Stay in Control

While Amplication streamlines the development process, you retain full ownership and control of your code. It's automation with autonomy.

Cost Savings

By reducing manual work, the risk of errors and associated costs decrease.

Efficient Data Integration with the DB Schema Import

Transitioning away from legacy systems doesn’t mean starting from scratch. With the DB Schema Import feature, you can effortlessly bring in your existing database schema, preserving the value of your legacy data and laying the foundation for new, modern applications.

Getting started with DB Schema Import

Ready to upgrade your legacy system? Here's a quick guide:

Generate your Prisma schema: Use Prisma's introspection to scan your legacy database and produce a schema.prisma file.
Upload to Amplication: Once your schema is ready, bring it into Amplication, which will automatically set up your entities, fields, and relations.
Pick your Database: Select either MySQL or PostgreSQL by installing the appropriate DB plugin.
Build & Commit: Generate and commit the code for your project via Amplication.

Check out our step-by-step guide for a deeper dive

A Real-Life Example

Let's dive into an example to demonstrate just how effortless it is to create a fully-functional service using this feature. In this walkthrough, we'll leverage a Prisma Schema file to automate the setup of entities, fields, and relations, laying the groundwork for a fully functioning application to manage events.

Create a new ‘Event Management’ service with Amplication
- Make sure to select MySQL or PostgreSQL database during the service creation wizard

After the service is created, click the “Create entities for my service” option

Within the entities page, click the “Upload Prisma Schema” button

Upload this sample Prisma schema file

datasource db {
  provider = "postgresql"
  url      = env("DB_URL")
}

generator client {
  provider = "prisma-client-js"
}

model Event {
    id          String      @id @default(uuid())
    name        String
    description String
    startDate   DateTime
    endDate     DateTime
    location    String
    attendees   Attendee[]
    sessions    Session[]
}

model Attendee {
    id          String      @id @default(uuid())
    name        String
    email       String     @unique
    eventId     String
    tikets      Ticket[]
    event       Event       @relation(fields: [eventId], references: [id])
}

model Ticket {
    id          String      @id @default(uuid())
    attendeeId  String
    ticketType  TicketType
    attendee    Attendee    @relation(fields: [attendeeId], references: [id])
}

model Session {
    id          String      @id @default(uuid())
    name        String
    speaker     String
    time        DateTime
    eventId     String
    event       Event       @relation(fields: [eventId], references: [id])
}

enum TicketType {
    FREE
    PAID
}

Watch the log as all the entities are imported.

Once the import is completed, click the entities tab to view all the newly created entities.

Switch to the ERD view to see a full picture of all the created entities with their fields and relations.

Make any additional configurations or changes, and when you are ready, build and commit the code to your Git repo.
Open the newly created PR to view the generated code for your Event Management service.

💡 Check out this Git repository showcasing the outcome of generating code with Amplication based on the example Prisma schema file

Gratitude to Our Beta Testers

A big thank you to the beta testers of this feature. Your invaluable feedback was instrumental in refining the DB Schema Import capability, making it more robust and developer-friendly.

In Conclusion

The challenge of updating old systems is significant, especially without losing data or running into issues. However, with Amplication's Database Schema Import, the process is simplified. This tool makes it easier to move from old systems to new ones without the typical headaches. Amplication offers an easy yet sophisticated solution to complex and time-consuming challenges as you look to modernize.

About Amplication

Amplication helps increase velocity and consistency for backend teams. It accelerates the development of microservices and backend applications using auto-generation of all the boilerplate and infrastructure code and allows developers to focus their efforts on developing the core business processes and logic.
Using an extensive plugin system, developers and platform teams can automate the generation of services while keeping all the best practices and know-how of the organization.

Try it now

You can start using Amplication by simply visiting https://app.amplication.com/

The Role of Queues in Building Efficient Distributed Applications

Yuval Hazaz — Thu, 10 Aug 2023 08:33:05 +0000

Introduction

When building mission-critical applications that manage large workloads, you must ensure our systems can handle the workloads. In some use cases, we can handle the demand with autoscaling policies on the infrastructure so that our infrastructure can scale up or down based on the demand.

However, there are certain use cases where this just isn't enough. For example, consider the following use case:

A user wishes to manage a content portfolio of all the articles a person has written.
This user then inputs a set of URLs (100 URLs) onto the system.
The system then visits these URLs and scrapes certain information needed to build the portfolio. This can include the: Article Name, Published Date, Outline, Reading Time, and Banner Image.
The system then stores it in an internal database.

A simple flowchart of this process is depicted below:

Figure: A flowchart of the content portfolio workflow

On its own, this process seems quite simple. But this can cause a lot of issues on a large scale:

The system cannot simultaneously handle thousands or millions of requests when each request may include hundreds of URLs.
We cannot predict if a URL is working or broken. Therefore, there's a high chance of system errors which may break the entire process and makes it very hard to manage retries.
The URL we visit may have high latency, slowing the overall process.

This approach will not scale and is not resilient enough to handle high demand.

How do queues work and help the system scale?

This is where messaging queues come into the picture.

A queue is a linear data structure that can process requests on a first-in-first-out basis. Queues play a huge role when we are working in distributed systems. Queues enable asynchronous communication within a distributed system. For example, consider the scenario shown below:

Figure: A synchronous workflow of the web scraper

The system will implement a linear-scaling model if we build the portfolio import scenario we discussed before using a synchronous process. This means that if URLInputLambda processes 100 invocations at the same time, the ScraperLambda will then be invoked synchronously to scrape the data, one after the other.

We wrongly assume that all requests will take the same time and succeed. But, when running into an error for a single URL, handling retries for that URL will be tough.

To resolve that, we should distribute the workflow and use a message queue to handle each task separately and asynchronously. It will look something like this:

Figure: Refactoring the architecture with a messaging queue

As shown above, we are using Apache Kafka as the messaging queue. Each URL will be pushed into the queue as a single event from the URLInputLambda. Next, the ScraperLambdaget each event from the queue to be processed.

By bringing this architectural change, we immediately gain these benefits:

The system is now responsive and highly scalable: Once the messages are fed into the queue, the URLInputLambda can complete the request processing and return a 200 status code to the user indicating that the messages have been accepted into the system. The queue can then take its own time and process the messages asynchronously. This means the forward-facing services can handle millions of requests without issues as the workflow is done asynchronously.
The system is now decoupled: By bringing the messaging queue, we have decoupled the system services. The two Lambda functions don't scale linearly, as the queue can manage the flow of requests. If needed, we can use more instances of the processor while keeping using minimal instances of other services.
The system is now resilient with improved fault tolerance: We can handle errors of broken URLs and set up a Dead Letter Queue, and push messages to the dead letter queue after X number of retries. This ensures no data is lost, and one failure does not disrupt the entire system, improving resiliency.

What are the different queueing strategies that are available for developers?

Well, now that we've got a rough idea of how powerful a queue is in a distributed system, it's essential to understand the different types of queues that we can use in our system:

Messaging Queues

We may use these standard queues for simple and linear use cases. Its responsibility is to store messages in a queue-based structure and let components send and receive messages from it.

Task Queues

A task queue is a specialized form of message queue that explicitly handles task distribution and processing. Consumers pick up tasks from the queue and execute them independently. This lets us integrate proper flow control, ensure our system scales non-linearly, and is often powered by event sourcing.

Publisher-Subscriber

This type of queueing strategy is sometimes treated as a building block of an event-driven architecture. It consists of a central publisher that pushes data onto subscribers who subscribe to it through an event-streaming-based approach. To learn more about using Pub/Sub models in microservices, look into this in-depth article.

The Pub/Sub strategy relies on event-driven architectures where publishers publish events for subscribers to process. One of the pitfalls of this approach is that it operates on a fire-and-forget basis, meaning that if a subscriber fails to process an event, that event gets lost forever. However, we can build a fault tolerance system around the subscriber to handle such errors.

When should we use queues in distributed applications?

This is one of our most important questions when working with a queue. "When should we use it?".

We should typically use a queue to accomplish some flow control or asynchronous processing in our application while enhancing application scalability, reliability, and resiliency.

Below are some best practices we should consider when using a queue.

Asynchronous processing

Queues let parts of our application perform operations asynchronously. For example, as discussed before, if you were to pass 100 URLs at a given time, and 1000 users performed this operation simultaneously, the system resources would be exhausted in one go as all of your time-consuming requests are immediately processed.

However, by introducing a queue, we can acknowledge back to the user with an "OK" message, while the system can take its time processing each request in an asynchronous nature without linearly scaling its resources. This lets you adopt architectural patterns such as "Queue Based Load Leveling".

Event-based communications

Queues are sometimes treated as event hubs as components can push data onto a queue, and other services can subscribe to it, poll for data, and begin processing. This lets us decouple parts of the application and enforce isolated, independent processing.

Decoupling microservices

Queues are used in microservices to decouple different services from each other. So, services can communicate through queues rather than calling each service directly.

This ensures better scalability of each service and reduces coupling between each service, thus, making it easier to modify or replace individual services without affecting others.

Improving fault tolerance and retries

Another reason to use a message queue is to ensure that messages are not lost. By default, queues can retry, or store failed messages through Dead Letter Queues. You can configure the Dead Letter Queue to let your system automatically push a failed message onto a special queue after X number of failed retries.

Delayed processing

Certain queueing services like Amazon SQS support delayed processing. This lets your queue send messages to its consumers by enforcing a delay of a set number of seconds. This is beneficial when you want your consumers to cope-up with the demand.

What are the common pitfalls and architectural challenges of queues?

But, messaging queues are not always the right soltuion for our application. It has its own set of pitfalls and challenges:

Fault Tolerance and Retries

Though we said fault tolerance is offered in queues, using it correctly can often be challenging. Most of the time, we will have to create a separate queue that will act as a dead letter queue, and we will have to configure it to work as we wish manually.

Idempotent Actions

Another challenge with queues is ensuring that a message is only processed once. By default, most queues use at-least-once delivery, with a potential for some messages to be handled more than once. So, it's essential to implement message deduplication using a deduplication ID that lets the queue prevent the message with the same deduplication ID from being pushed to a consumer over a certain period.

What are some queueing services that we can use?

Below are some popular queue services and technologies we can use

1. RabbitMQ

RabbitMQ is a robust and highly configurable open-source message broker that implements the Advanced Message Queuing Protocol (AMQP).

Pros:

Supports various messaging patterns, including point-to-point, publish-subscribe, request-reply, and more.
Rich feature set with advanced routing, message acknowledgment, message durability, and priority queues.
High reliability and fault tolerance with support for clustering and message replication.
Integrates well with multiple programming languages and platforms.

Cons:

Slightly complex setup and configuration compared to other queuing systems.

2. Apache Kafka:

Apache Kafka is a distributed event streaming platform designed for high-throughput, real-time data processing and event-driven architectures.

Pros:

Scalable and high throughput with support for parallel processing and partitioning.
Provides strong durability and fault tolerance through data replication across brokers.
Real-time data streaming and processing capabilities.
Supports event replay, allowing consumers to go back and reprocess past events.
Efficiently handles large-scale data streams and has low-latency characteristics.

Cons:

Slightly more complex to set up and manage compared to traditional message queues.
It may not be the best fit for point-to-point messaging or request-reply patterns.

3. Amazon SQS

Amazon SQS is a fully managed message queue service that provides a reliable and scalable solution for asynchronous messaging between distributed components and microservices.

Pros:

Fully managed service with high availability, durability, and automatic scaling.
Supports two types of queues: Standard Queue for high throughput and FIFO Queue for ordered, exactly-once processing.
Super easy deduplication rules configuration for FIFO queues.
Support for dead-letter queues.

Cons:

Messages are limited to 256KB in size; anything bigger will require implementing a solution that leverages S3 buckets.
Limitations on the number of in-flight messages and FIFO throughput.

4. Google Pub/Sub:

Google Pub/Sub is a fully managed, highly scalable messaging service that enables real-time messaging and event-driven architectures.

Pros:

Scales automatically based on demand and offers low-latency message delivery.
Offers fine-grained access controls for securing messages and topics.

Cons:

Limited features compared to more advanced event streaming platforms like Apache Kafka.

Wrapping up

Messaging queues have become the industry standard for building highly scalable, reliable, available, and resilient distributed systems. Their ability to decouple components, handle asynchronous processing, and provide fault tolerance has made them vital to modern application architectures.

However, it's essential to understand the challenges associated with messaging queues. Proper handling of fault tolerance, idempotent actions, and message deduplication is crucial to avoid potential pitfalls.

Considering these aspects, we can build systems that efficiently handle complex and demanding workloads!

How Amplication Fits in

Adopting asynchronous communication models can improve your microservices architecture's scalability, reliability, and performance. But building a scalable microservice architecture requires a lot of planning and boilerplate, scaffolding, and repetitive coding.

Amplication is a code generator for backend services that generates all the repetitive parts of microservices architecture, including communication between services using message brokers with all the best practices and industry standards.

You can build the foundation of your backend services with Amplication in minutes and focus on the business value of your product.

Node.js asynchronous flow control and event loop

Yuval Hazaz — Tue, 08 Aug 2023 05:06:28 +0000

Node.js's powerful asynchronous, event-driven architecture has revolutionized server-side development, enabling the creation of highly scalable and efficient applications through non-blocking I/O operations. However, grasping the inner workings of Node.js's asynchronous flow control and event loop can be quite challenging for newcomers and seasoned developers alike. In this article, we will explore the asynchronous nature of Node.js, including its core components like the Event Loop, Asynchronous APIs, and the Node.js Call Stack.

What is asynchronous flow in Node.js?

Asynchronous flow refers to the way Node.js handles and executes ensuring that the main program flow remains unblocked. As a server-side runtime environment built on Chrome's V8 JavaScript engine, Node.js efficiently manages concurrent tasks and optimizes resource utilization. It achieves this by delegating many operations, such as file I/O, network requests, and database queries, to separate background threads, enabling the main thread to proceed with other tasks. Upon completion, the results from these background tasks are returned to the main thread, often using callbacks, promises, or async/await mechanisms. This approach allows Node.js to maintain responsiveness and scalability, making it a preferred choice for building non-blocking, high-performance applications that can handle multiple concurrent operations effectively.

At the heart of Node.js's asynchronous flow is the event loop, a crucial component that plays a vital role in managing and executing tasks efficiently. The event loop is responsible for efficiently scheduling and executing asynchronous tasks. It constantly monitors the task queue, executing pending operations when the main thread becomes idle, further enhancing Node.js's responsiveness and enabling the seamless handling of concurrent tasks. Now, let's delve into the specifics of the Node.js event loop and understand how it drives the asynchronous flow of Node.js.

Event loop in Node.js

The Event Loop constitutes a vital aspect of Node.js, enabling it to manage asynchronous operations efficiently. It maintains the application's responsiveness by continuously monitoring the Event Queue for pending events. Node.js's Event Loop follows a straightforward yet highly effective mechanism.

Event registration: Whenever an asynchronous operation is initiated, such as reading a file or making a network request, the corresponding event is registered and added to the Event Queue.
Event loop execution: The Event Loop perpetually checks the Event Queue for pending events. Upon completion of an event, it is dequeued, and its associated callback is added to the Node.js Call Stack for execution.
Callback execution: The callbacks associated with the dequeued events are executed, enabling the application to respond to the events.
Non-blocking execution: Node.js's Asynchronous APIs ensure that while waiting for an operation to complete, the application can continue executing other tasks, making it highly performant for I/O-intensive operations.

Source: https://www.geeksforgeeks.org/node-js-event-loop/

The above figure depicts an example of a Node.js event loop. Upon Node.js startup, the event loop is initialized, and the input script is processed. This input script might involve asynchronous API calls and the scheduling of timers.

Node.js utilizes a dedicated library module called libuv to handle asynchronous operations. This module, in conjunction with Node's underlying logic, manages a specialized thread pool known as the libuv thread pool. The libuv thread pool consists of four threads by default which are responsible for offloading tasks that are too resource-intensive for the event loop. Such tasks encompass I/O operations, opening and closing connections, and handling setTimeouts.

When the libuv thread pool completes a task, a corresponding callback function is invoked. This callback function takes care of any potential errors and performs other necessary operations. Subsequently, the callback function is added to the event queue. As the call stack becomes empty, events from the event queue are processed, allowing the callbacks to be executed by placing them onto the call stack.

Node.js event loop comprises multiple phases, with each phase dedicated to a particular task. The below diagram shows a simplified overview of the different phases in the event loop.

Source: https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick

As the above diagram depicts there are different phases in the Node.js event loop to handle different types of operations.

Timers: Timers phase executes callbacks scheduled by setTimeout() and setInterval(). These callbacks will be triggered as soon as possible after the specified amount of time has elapsed. However, external factors like Operating System scheduling or the execution of other callbacks may introduce delays in their execution.
Pending callbacks: This phase is dedicated to executing callbacks for specific system operations, like handling TCP errors.
Idle, prepare: Idle phase is only used internally. In this phase, the event loop is not actively processing tasks, providing an opportunity to perform background operations like garbage collection.
Poll: Two main functions are performed during the poll phase, calculating the appropriate duration for blocking and polling I/O, and processing events in the poll queue. When the event loop enters the poll phase without any scheduled timers, it checks the poll queue. If the queue contains callbacks, they are executed synchronously until the queue is empty or reaches a system-dependent limit. In the absence of callbacks in the poll queue, the event loop proceeds to either the check phase if setImmediate() scripts are scheduled or waits for new callbacks to be added to the queue, executing them immediately. Once the poll queue becomes empty, the event loop checks if any timers have reached their time thresholds and, if so, moves back to the timers phase to execute their respective callbacks.
Check: The check phase invokes any setImmediate() callbacks that have been added to the queue. As the code is executed, the event loop will eventually reach the poll phase. However, if a callback has been scheduled using setImmediate() and the poll phase becomes idle, the event loop will proceed directly to the check phase instead of waiting for poll events to occur.
Close callbacks: When a socket or handle is closed suddenly, the close event is emitted in this phase. However, if the closure is not immediate, the closeevent will be emitted using process.nextTick().

Understanding Call Stack and Asynchronous APIs

For a comprehensive understanding of Node.js's asynchronous flow control, it is essential to comprehend the Node.js Call Stack and its interaction with asynchronous APIs.

The Call Stack functions as a data structure, keeping track of function calls within the program. When a function is invoked, it is added to the top of the stack, and upon completion, it is removed, following a last-in-first-out (LIFO) order. As shown in the diagram below Node.js will initially create a global execution context for the script and place it at the bottom of the stack, and then it will create a function execution context for each function called and place it on the stack. This execution stack is also known as the Call Stack.

Source: https://www.javatpoint.com/javascript-call-stack

Furthermore, as Node.js is designed to operate asynchronously, it provides many APIs that use callbacks or promises to manage the results of asynchronous operations. When an asynchronous function is invoked, it is offloaded to the Node.js runtime environment, allowing the Event Loop to continue processing other tasks. Once the asynchronous operation completes, the associated callback is placed in the Callback Queue, awaiting the Event Loop's attention for execution. When the Call Stack is empty, the Event Loop picks the first callback from the Callback Queue and pushes it onto the Call Stack for execution. This approach ensures that asynchronous tasks do not block the main thread, contributing to the application's responsiveness.

Benefits of Asynchronous Programming in Node.js

The asynchronous nature of Node.js brings forth several notable advantages that benefit developers and the applications they build such as,

Scalability and Concurrency: Node.js's asynchronous nature allows it to handle a large number of concurrent connections efficiently. By leveraging non-blocking I/O operations and asynchronous event handling, Node.js can serve multiple clients simultaneously without consuming excessive resources.
Resource Efficiency: Node.js utilizes a single-threaded event loop to handle multiple concurrent connections, reducing the overhead of creating and managing threads for each connection. This approach results in better memory utilization and improved resource efficiency.
Improved Responsiveness: Asynchronous operations prevent the application from becoming unresponsive during time-consuming tasks, leading to enhanced user experiences.
Simplified Code: The asynchronous model allows developers to write clean and concise code, avoiding complex control flow and the notorious "callback hell." Asynchronous APIs, along with the adoption of promises and async/await, promote more readable and maintainable codebases.
Easier Debugging: Asynchronous operations in Node.js are designed to provide meaningful error messages, making identifying and troubleshooting issues easier.

Common Pitfalls and How to Avoid Them

While the asynchronous nature offers remarkable benefits, it also introduces certain challenges that developers should be mindful of. Here are some common pitfalls and recommended strategies to overcome them:

Blocking the event loop: Running CPU-intensive tasks can cause the event loop becoming blocked to other incoming events or callbacks causing slow application performance, reduced concurrency, and a negative user experience. By utilizing asynchronous APIs and non-blocking I/O operations, enabling tasks to be delegated to the background while the event loop continues processing other events efficiently can maintain a responsive event loop.
Callback hell: Chaining multiple callbacks can lead to deeply nested and hard-to-read code. Adopting promises or async/await can significantly improve code readability and maintainability.
Uncaught exceptions: Unhandled errors in asynchronous operations can crash the application. Always implement proper error-handling mechanisms to gracefully handle exceptions and prevent application failures.
Memory leaks: Improper management of event listeners can result in memory leaks. Ensure to remove event listeners when they are no longer needed to prevent unnecessary memory consumption.
Overuse of asynchronous operations: Not all operations need to be asynchronous. Carefully choose synchronous and asynchronous operations to strike the right balance between performance and code clarity.

Conclusion

Asynchronous programming is a powerful paradigm that efficiently handles large-scale, concurrent operations. Node.js heavily relies on this approach to achieve remarkable concurrency and scalability. Its asynchronous nature, centered around the Event Loop, Asynchronous APIs, and the Node.js Call Stack, enables efficient management of asynchronous operations, delivering highly responsive applications.

By embracing the benefits of asynchronous flow, developers can create high-performance and scalable applications that respond to events in real-time, providing users with a seamless and efficient experience. However, it is crucial to remain mindful of potential pitfalls, like blocking the event loop or getting lost in callback hell, and to adopt best practices to ensure smooth execution and error handling. Overall, Node.js's asynchronous nature offers a robust foundation for building modern, responsive server-side applications.

Node.js and Amplication

Amplication can automatically generate fully functional services based on TypeScript and Node.js to speed up your development process.

Furthermore, Amplication can include technologies like NestJS, Prisma, PostgreSQL, MySQL, MongoDB, Passport, Jest, and Docker in the generated services. Hence, you can automatically create database connections, authentication and authorization features, unit tests, and ORMs for your Node.js service.

You can find the getting started guide here.

Create API with GraphQL, Prisma, and MongoDB

Yuval Hazaz — Thu, 13 Jul 2023 09:23:13 +0000

GraphQL, Prisma, and MongoDB have become the go-to options for building highly scalable APIs for modern web applications. With GraphQL, developers can easily define and request the precise data they need, while Prisma and MongoDB simplify database interactions.

This article aims to provide a comprehensive guide to building a robust API with GraphQL, Prisma, MongoDB, and Node.js. But, I will take a different path than the traditional approach and introduce Amplication to demonstrate how to further simplify the process of creating a GraphQL API.

GraphQL

GraphQL is a server-side runtime and a query language for APIs. It allows developers to define queries that define the exact structure of the data they require, eliminating the over-fetching and under-fetching of data. Hence, it has become popular over REST APIs in modern application development.

Prisma

Prisma is an open-source ORM that streamlines database access in your applications. It offers a range of tools and features to simplify database access by abstracting away the database-specific complexities. This allows developers to seamlessly interact with multiple database systems like PostgreSQL, MySQL, SQLite, and SQL Server without writing specific code tailored to each database.

MongoDB

MongoDB is a highly scalable document-type database. Its flexible schema approach allows developers to adapt and modify the data structure as needed, providing agility in application development.

Amplication

Amplication is an open-source platform that automatically generates APIs and clients based on pre-defined data models. It streamlines the development process by minimizing repetitive coding tasks and boilerplate code, enabling developers to focus on building backend services more efficiently.

How to Create API with GraphQL, Prisma, and MongoDB

To demonstrate a combination of these technologies, I have used a small inventory management process to implement. The scenario will contain parts and packets where a packet contains parts.

Step 1 - Creating a project with Amplication

The first step is creating a project with Amplication. You will see an option to create a New project in the bottom left corner of the Amplication dashboard. It will open a modal to enter a project name, and click the Create new Project button.

Step 2 - Create and setup a service and connect it to the Git repository

Click on Add Resource button and select the Service option from the dropdown. It will redirect you to a page to enter a name for the service.

Then, you will be asked to connect a git repository.

Next, you need to to choose between GraphQL and REST. In this case, I will choose GraphQL.

Next, Amplication gives us two options to build the project structure. If you plan to create multiple services in the repo, choose the Monorepo option. In this case, we only have GraphQL API service to create. So, I will be selecting a Polyrepo here.

Then you have to choose the database option. In this project, we are using MongoDB as the database provider.

Then, it will ask you to select a template to generate entities automatically. In this case, we will define the data model later, so select "Empty".

Finally, you can include authentication for your service by selecting the Include Auth Module option. It will automatically generate the authentication code for your service.

Step 3 - Defining the data model with Amplication

Amplication is using Prisma for the ORM. Prisma simplifies database operations and integrates seamlessly with GraphQL. In this section, we will define our data model using Amplication, and it will generate models with Prisma's declarative syntax.

You can create new entities by clicking the Add Entity button in the Entities tab. Here, I'm creating two entities that have a relation with each other.

Part model - This consists of the fields - id, description, weight, color, created time, and last modified time.

Packet model - This consists of the fields - id, name, parts, created time, and last modified time.

You can also define the relationship between the two entities through Amplication. For that, open the Part entity, and create a field named "packet". It will automatically be set as a relation field. Navigate back to the "Packet" entity, and you can see that you also have a new relation field called "Parts".

One Packet can be related to many Parts.

One Part can be related to one Packet.

Step 4: Commit changes

Once all the entities are created, click the Commit changes & build button to sync the changes with the GitHub repository.

Step 5: Get the source code

Navigate to the GitHub repo by clicking the Open With GitHub button and clone the source code to your local machine.

Amplication has successfully generated all the required files and boilerplate code for you. For example, prisma/prisma.schema file contains the data models we created from the Amplication UI.

datasource mongo {
  provider = "mongodb"
  url      = env("DB_URL")
}

generator client {
  provider = "prisma-client-js"
}

model User {
  createdAt DateTime @default(now())
  firstName String?
  id        String   @id @default(auto()) @map("_id") @mongo.ObjectId
  lastName  String?
  password  String
  roles     Json
  updatedAt DateTime @updatedAt
  username  String   @unique
}

model Part {
  color       String?
  createdAt   DateTime @default(now())
  description String?
  id          String   @id @default(auto()) @map("_id") @mongo.ObjectId
  packet      Packet?  @relation(fields: [packetId], references: [id])
  packetId    String?
  updatedAt   DateTime @updatedAt
  weight      String?
}

model Packet {
  createdAt DateTime @default(now())
  id        String   @id @default(auto()) @map("_id") @mongo.ObjectId
  name      String?
  parts     Part[]
  updatedAt DateTime @updatedAt
}

Now, you can customize the application based on your specific requirements.

Step 6: Install npm packages

Once the application is ready, you need to install npm packages and dependencies using npm install command.

npm i

Step 7: Start the Docker container for the database

Start the Docker container to run the database using the below command.

npm run docker:db

Then create the application schema on the database using Prisma. You can use the below migration commands for that:

npm run prisma:generate 
npm run db:init

Step 8: Run the application

Finally, you can run the application using npm run start command. It will start the server at http://localhost:3000

npm run start

Also, you can access the GraphQL API using the GraphQL Playground available at http://localhost:3000/graphql.

Note: All the above instructions are commands are also listed in the README.md file in the root of the server code

Step 9: Testing the APIs using GraphQL playground

Once the GraphQL server is up, you can test your APIs with the GraphQL Playground. The below examples show how to test the GraphQL APIs for creating a Part and retrieving all parts. Now, you can customize the application based on your specific requirements.

Sample Mutation API

mutation {
  createPart(data:{
    color: "orange"
    description: "orange part 001"
    weight: "25"
  }) {
    color
    createdAt
    description
    id
    updatedAt
    weight
  }
}

{
    "data": {
        "createPart": {
            "color": "orange",
            "createdAt": "2023-06-19T06:17:48.964Z",
            "description": "orange part 001",
            "id": "648ff30c6af532b09aeb5562",
            "updatedAt": "2023-06-19T06:17:48.964Z",
            "weight": 25
        }
    }
}

Sample Query API


query{
    parts{
        color
        createdAt        
        description
        id
        updatedAt
        weight
        packet{
             id
         }
    }
}

{
    "data": {
        "parts": [
            {
                "color": "red",
                "createdAt": "2023-06-13T20:22:08.544Z",
                "description": "red part 001",
                "id": "6488cff0c0b65f41f03c4d3a",
                "updatedAt": "2023-06-13T20:22:08.544Z",
                "weight": 20,
                "packet": null
            },
            {
                "color": "green",
                "createdAt": "2023-06-13T20:22:26.267Z",
                "description": "green part 001",
                "id": "6488d002c0b65f41f03c4d3b",
                "updatedAt": "2023-06-13T20:22:26.267Z",
                "weight": 50,
                "packet": null
            },
            {
                "color": "yellow",
                "createdAt": "2023-06-13T20:25:05.531Z",
                "description": "yellow part 001",
                "id": "6488d0a1c0b65f41f03c4d3d",
                "updatedAt": "2023-06-13T20:25:05.531Z",
                "weight": 50,
                "packet": null
            },
        ]
    }
}

Conclusion

As you can see, Amplication significantly simplifies the process of creating GraphQL APIs while reducing the effort you need to initialize a new project. You can effortlessly generate all the required files for multiple data models with a few straightforward steps.

Amplication is an open-source tool designed to streamline Node.js development by generating fully functional Node.js services. It goes beyond supporting GraphQL and MongoDB. With Amplication, you can create Node.js APIs and services using other technologies like PostgreSQL, MySQL, Passport, Jest, and Docker. I strongly encourage you to give Amplication a try and witness the impact it can have on your development workflow.

How to Build a Node.js GraphQL API With NestJS and Prisma

Yuval Hazaz — Thu, 06 Jul 2023 07:59:11 +0000

Are you interested in building highly scalable and performant server-side applications? Have you considered using Node.js, a widely used and robust runtime environment? Recently, developers have preferred using GraphQL with Node.js to build faster and more flexible APIs than the more traditional REST APIs.

To fully leverage the potential of GraphQL, it is essential to have a reliable framework that can handle the complexities of a growing codebase and an Object Relational Mapping (ORM) tool that simplifies database interactions. NestJS and Prisma are a great couple that can provide exactly that. NestJS is a powerful framework specifically designed for building Node.js applications, while Prisma is an ORM tool that provides a type-safe API for querying databases in Node.js.

This article aims to provide insights on building a Node.js GraphQL API using NestJS and Prisma while addressing some of the most frequently asked questions about these technologies.

Many of you are likely well-versed in creating traditional Node.js applications. However, this article will present a unique approach to building a Node.js GraphQL API with NestJS and Prisma, utilizing the Amplication platform to simplify the development process further. But before getting into action, let's get familiar with the technologies mentioned above.

GraphQL

GraphQL is an open-source query language that provides a more efficient and flexible way to request and manipulate server data. In REST APIs, users must send multiple requests to different endpoints to retrieve various data. In contrast, GraphQL allows users to construct queries that define the exact shape and structure of the data they require, eliminating the over-fetching and under-fetching of data.

Prisma

Prisma is an open-source database toolkit and ORM that provides tools and features to simplify database access in your applications. It abstracts the database-specific details, allowing developers to work with multiple database systems (such as PostgreSQL, MySQL, SQLite, and SQL Server) without writing specific code for each database.

NestJS

NestJS is an open-source framework for building efficient, scalable, and maintainable server-side applications using TypeScript. Angular inspires NestJS and uses TypeScript features like strong typing, decorators, and dependency injection to provide a robust architecture for building server-side applications.

Amplication

In this article, I will combine the power of Amplication, Prisma, GraphQL, and NestJS to create a simple Node.js service with three entities to manage my Blogs.

The Steps

Step 1 - Create a New Project in Amplication

Once you log in to the Amplication dashboard, you will see an option to create a New project in the bottom left corner. It will open a modal to enter a project name and click the Create new Project button.

Step 2 - Create a New Service

Then, click the Add Resource button and select the Service option from the dropdown.

It will redirect you to a new wizard to configure the new service. First, you need to enter a name for the service.

Then, connect to a GitHub repository where you want to get the generated code.

Next, you need to choose between GraphQL and REST. Since this article is about GraphQL, I have only enabled GraphQL API and Admin UI options.

Next, you can select between Monorepo and Polyrepo based on your project and team requirements. For this example, you can leave the default settings as it is.

Next, you must select between PostgreSQL, MongoDB, and MySQL for database options.

Also, Amplication can automatically generate entities for your database models if you prefer. We will define the data model for our use case later, so select "Empty."

Finally, you can include authentication for your service. If you opt to have the auth module, Amplication will automatically generate the authentication code for your service.

Once the service is created, you will see a window like the one below. Click on Create entities for my service option to start creating entities for the new service.

Step 3 - Create Entities

By default, Amplication creates a user entity to manage users related to your service. You can easily create new entities by clicking the Add Entity button in the Entities tab.

First, you need to enter the entity name.

Then, you will get a window like the one below where you can create fields for the entity. For each field, you can configure properties like uniqueness, required or not, searchability, data type, max length, etc.

After creating the models, you can access the Amplication UI and review them to ensure they have been generated correctly per your expectations. As explained earlier, I have created three entities for my BlogService: Blog, Publication, and Author.

Step 4: Commit Changes

Once all the entities are created, click the Commit changes & build button to sync the changes with the GitHub repository.

Step 5: Get the Source Code

Now you can navigate to the GitHub repo by clicking the Open With GitHub button and clone the source code to your local machine.

That concludes the process. Amplication has successfully generated all the required files and boilerplate code for you. For example, the amplication-example/apps/blog-service/src/blog/base folder contains the Blog model, DTOs, GraphQL resolver, Service, and tests.

Now you can open the code with VSCode, and customize it.

Step 6: Install npm Packages

Once the application is ready, you must install npm packages and dependencies using the npm install command.

Step 7: Start the Docker Container for the Database

Start the Docker container to run the database using the below command.

npm run docker:db

Then create the application schema on the database using Prisma. You can use the below migration commands for that:

npm run prisma:generate 
npm run db:init

Step 8: Run the Application

Finally, you can run the application using the npm run start command. It will start the server at http://localhost:3000.

Also, you can access the GraphQL server through http://localhost:3000/graphql.

Fun Fact

One fascinating fact about the Amplication blog is that its backend is built entirely using Amplication itself. By utilizing Amplication for our own blog, we are not only showcasing its potential but also benefiting from its efficiency and productivity-enhancing features. It's worth mentioning that the entire codebase of the Amplication blog's backend is publicly available on GitHub, allowing developers to explore, learn, and contribute to its development. You can find the code repository for the blog server at https://github.com/amplication/blog-server. We believe in transparency and collaboration, and making the code available underscores our commitment to fostering an open-source community. Feel free to delve into the codebase and witness firsthand how Amplication powers the backend of our blog.

Conclusion

In summary, Amplication dramatically simplifies generating GraphQL APIs using Nest.JS and Prisma. With just a few steps, developers can quickly generate all the necessary files for multiple data models.

Amplication is a free, open-source tool that accelerates development by creating fully functional Node.js services. In addition to Nest.js and Prisma, it supports several other technologies such as PostgreSQL, MySQL, MongoDB, Passport, Jest, and Docker. Therefore, I encourage you to try Amplication and experience the difference it can make in your development workflow.

FAQs

Q1: What is Prisma?

Prisma is a widely recognized ORM (Object-Relational Mapping) tool that smoothly integrates with NestJS. Using Prisma, developers can easily create database schemas with a simple syntax and construct type-safe APIs for querying databases. Additionally, Prisma facilitates writing database queries and resolving database-related errors in NestJS applications, making it an efficient tool for developers.

Q2: Does NestJS use GraphQL?

Indeed, NestJS has built-in support for GraphQL. It provides two methods for constructing applications with GraphQL: code-first and schema-first. The code-first approach employs decorators and TypeScript classes to produce a GraphQL schema, whereas the schema-first approach utilizes GraphQL SDL (Schema Definition Language).

Q3: How Is GraphQL Different From REST?

GraphQL and REST are 2 of the most popular approaches in building APIs. However, there are some significant differences between them. You can find a detailed comparison between GraphQL and REST here.

What is Open-source Software: Everything you wanted to know

Yuval Hazaz — Tue, 30 May 2023 05:52:38 +0000

In the dynamic realm of technology, where innovation reigns supreme, one approach has gained remarkable traction—open-source software. With its ethos of transparency, collaboration, and limitless possibilities, open-source software has revolutionized the way we develop, share, and customize applications. From operating systems to data analysis tools, its impact spans across diverse domains. Join us on a journey as we delve into the history, advantages, and vibrant ecosystem of open-source software, uncovering how it empowers developers and businesses alike. Discover why open-source software is not just a buzzword but a transformative force shaping the digital landscape.

Open-source software (OSS) is code distributed under a license that allows the users to access the software's source code and modify and build new features upon the existing software to meet the user requirements. Open-source software applications are vast and varied, ranging from operating systems, web browsers, and mobile applications to data analysis tools, machine learning frameworks, and cloud computing platforms. This article will discuss the open-source software model, history, and advantages of using OSS. We will also examine the difference between open-source software and closed-source software.

History of Open-Source Software

In earlier days, academics and corporate researchers working in collaboration primarily implemented software. Therefore, everybody shared software often due to the openness and cooperation already established in academia. However, by the early 1970s, software development had become more expensive, and corporations started licensing and selling software products. IBM was one of the market leaders at the time. By the late 1970s and early 1980s, software vendors began regularly charging for software applications and licensing and restricting new software development on the existing platforms. Furthermore, vendors began to distribute only the machine executables of the software without the source code.

Among the many developers unhappy with these developments was Richard Stallman, the founder of the Free Software Foundation, who initiated the GNU project intending to build a complete, free operating system. Furthermore, they created the GNU General Public License (GPL), allowing users to copy, modify and redistribute the software for free as long as their new version is distributed under the same license.

With the rise of the Internet and software built around it, collaborative development became much easier, and open-source software had massive growth. By 1991 Linus Torvalds announced the project to create an operating system kernel, and he released the first version under the name Linux in 1994. Linux gained much traction since it was a free and open-source alternative to the other proprietary operating systems.
The introduction of the Apache web server in 1995 quickly made it one of the most popular web servers in the world, enabling developers to host their web applications without any cost for the web server and yet be used by millions of websites and web applications. In addition, the open-source license of the Apache web servers enabled the developers to manipulate the source code to improve its functionality.

Another powerful open-source software was the MySQL database, introduced in 1995. MySQL is among the most widely used databases, which enables developers to modify the source code to improve performance and reliability freely.

In 1998, Open Source Initiative was founded to promote and support open-source software, paving the way to the current open-source software ecosystem as we know it. In addition, open-source software enabled the innovation of new technologies based on open-source software, with the collaboration of a massive open-source community. Amplication is one such example of an innovative open-source tool that can help developers build high-quality applications efficiently. In essence, open-source software became one of the factors influencing the expansion of the Internet.

Open-Source Software Development Model

The open-source software model is a collaborative approach to building software. Contributors can be anyone from individual developers to large corporations. Developers contribute to the project via collaborative platforms like GitHub, and all contributors are allowed to modify the source code in the repository as long as the modifications follow the license terms and software standards.

When creating a new project, all contributors have access to add new features, modify the source code, and create pull requests to make the software better. The project maintainers review these pull requests and merge them into the repository. Then, the codebase maintainers can create a new release with the newly added features.

The open-source software model relies on the community to build and improve the software with feedback. But some companies also specialize in building open-source software, such as Red Hat, Canonical, SUSE, Docker, and HashiCorp.

Open Source Vs. Closed Source Software

Open and closed sources are software models that differ in licensing, source code availability, and many other factors.

Source code and development

In the open-source software model, source code is freely accessible and developed by a large community of developers. Whereas in closed-source software, the source code is confidential, hidden from the users, and maintained by the owning company.

Licensing

Open-source software is distributed under an open-source license such as GNU General Public License, which makes it accessible for anyone to modify and distribute under the same license. But in closed-source software, they are distributed under a proprietary license which restricts any unauthorized modifications or distribution of the software.

Maintenance

The community maintains open-source software and decides the required features and the road map according to their needs. On the other hand, closed-source software is based on proprietary models where the owner corporation owns and manages the software's features and road map.

Features, support, and cost

Open-source software provides flexibility and customization since users can modify the source code. But more support might be needed when a unique issue arises. On the other hand, whereas closed software is expensive, it provides support for the software. Therefore, it may have advanced features compared to its open-source counterparts.

There are both pros and cons to both software models. Therefore, choosing the correct software model depends on the specific use case.

Benefits of Open-Source Software

Open-source software provides many benefits, such as:

Reduced expenses

Most open software is free and significantly less expensive than commercial software alternatives. Therefore, small businesses and start-ups can benefit considerably from open-source software.

Customizability

Open-source software enables users to modify the software as needed since the source code is freely accessible. Therefore, open-source software is an excellent fit for businesses that need custom software tailored to their specific needs without reinventing the wheel.

Rapid and innovative development

Since a large community creates open-source software, it can result in quick and innovative development cycles because the software is built via teamwork.

Transparency and security

Since OSS is developed through a collaborative process involving a vast community, any security flaws or defects in the program can be found quickly. Moreover, the transparency of the source code can be easily verified because the source code is publicly available.

Overall, OSS provides many benefits ranging from cost savings to transparency and security of the software. In addition, by leveraging the knowledge and expertise of a large global community, OSS can be developed much more efficiently while serving a broader range of perspectives.

Applications of Open-Source Software

OSS has a wide range of applications, including:

Operating systems: Open-source operating systems like Linux provide an excellent alternative to their commercial counterparts like Windows.
Web servers: Many web servers like Apache and Nginx are OSS and highly used to build websites and web applications.
Database Management: Database software is essential to building applications, and many popular open-source alternatives are available for storing and managing data, such as MySQL and PostgreSQL.
Development tools: Several tools are utilized when implementing software, and many popular open-source development tools are available to ease the development process, such as Git and languages like Python. In particular, Amplication is a powerful open-source development tool that can help streamline development. If you're a developer looking to accelerate your software development process, consider checking out Amplication and contributing to the growing open-source community.
Security: OpenSSL and OpenSSH are highly used open-source software in secure communication and data encryption.

Open Source and Amplication

Amplication is an open-source tool for developing software that anyone can access, use, edit, and distribute under the rules of the license. Amplication's open-source nature also makes it a transparent and collaborative tool. Users can audit the source code and help identify and fix any bugs or security issues found in the source code. And by utilizing the power of the open-source nature, Amplication can benefit from a wide range of perspectives and expertise, resulting in higher-quality code and features. Amplication's open-source nature makes it a powerful and accessible tool for developers who want to build and deploy scalable web applications quickly and efficiently. It also aligns with the philosophy of open-source software, which emphasizes collaboration, transparency, and accessibility.

If you like the work we do, consider giving us a 🌟 on GitHub.

Conclusion

Open-source software is a valuable and vital part of the technology ecosystem. Since the dawn of the Internet era, open-source software has contributed significantly to shaping the development and innovation of technologies. Therefore, though issues need to be addressed, open-source software continues to gain ground as an attractive alternative to commercialize software for small businesses or anyone looking to develop tailor-made software for their specific needs. Therefore, irrespective of your role, it is essential to understand the basics of open-source software to make informed decisions when choosing software for your business or personal life.

I hope you have found this article helpful. Thank you!

FAQ

Q1: What's the difference between open-source and free software?

The difference between open source and free software lies in the philosophies. Free software advocates user freedom and ethical software-related considerations, in contrast to open-source software, which focuses mainly on the collaborative development of software.
Hence, open source is a way of building software, whereas free software is a social movement.

Q2: What is an example of free and open-source software?

The Linux operating system is a free and open-source software. Other software includes LibreOffice, Mozilla Firefox, VLC Media Player, Apache web server, Python, and many more.

Q3: What is an example of open-source software?

OpenOffice is an open-source alternative to Microsoft Office. Many other open-source tools exist, such as Amplication, Git, and WordPress.

Q4: Where is open source used?

Open-source software is used in various industries and domains, such as software development, web servers, cloud computing, multimedia, education, etc. In addition, many developers use open-source tools and libraries to build software products.

7 Tips to Build Scalable Node.js Applications

Yuval Hazaz — Tue, 16 May 2023 06:54:21 +0000

Scaling should not be an afterthought when it comes to building software. As the number of users of an application increases, the application should scale and handle the increased payloads effectively.

Many technologies can be used to build such scalable applications, and Node.js is a popular choice among the rest. Node.js is a JavaScript framework created on Chrome's V8 JavaScript engine and, if utilized correctly, can be used to build highly scalable mission-critical applications. This article will discuss several tips which can be helpful when it comes to building scalable applications using Node.js.

1. Worker threads & concurrency

Node.js executes JavaScript code in a single-threaded model. However, Node.js can function as a multithreaded framework by utilizing the libuv C library to create hidden threads (see the event loop) which handle I/O operations, and network requests asynchronously. But, CPU-intensive tasks such as image or video processing can block the event loop and prevent subsequent requests from executing, increasing the application's latency.

Therefore, to handle such scenarios, worker threads were introduced in Node.js v10 as an experimental feature and a stable version was released in Node.js v12.

How Does The Worker Thread Work?

A worker thread is an execution thread within a Node.js process with an isolated environment consisting of an event loop. This ensures it can run parallel with other threads to perform expensive operations without blocking the main event loop.

The parent thread creates worker threads to execute resource-intensive tasks isolated from other threads. This ensures that the parent thread operates smoothly without blocking any operations.

Creating a worker thread is as simple as importing the worker_threads library and creating a new object.

const { Worker, isMainThread, parentPort } = require('worker_threads');

if (isMainThread) {
  // Main thread
  console.log('Starting worker threads...');

  // Start a worker thread
  const worker1 = new Worker(__filename);

  // Send data to the worker threads
  worker1.postMessage({ id: 1, start: 0, end: 99 });

  // Listen for messages from the worker threads
  worker1.on('message', (message) => {
    console.log(`Main thread received message from worker ${message.id}: ${message.result}`);
  });
} else {
  // Worker thread
  parentPort.on('message', (data) => {
    console.log(`Worker ${data.id} received data: start=${data.start}, end=${data.end}`);

    // Perform a computationally expensive task
    let result = 0;
    for (let i = data.start; i <= data.end; i++) {
      result += i;
    }

    // Send the result back to the main thread
    parentPort.postMessage({ id: data.id, result });
  });
}

The snippet above depicts a real example of a worker thread in Node.js.

The primary (parent) thread creates a worker thread using the code in the same file and then passes the data to the worker using the message channel. Hereafter, the worker thread executes the assigned task using the data sent via the message channel. After the expensive operation has finished its execution in the worker thread, the result is sent back to the main thread, where a callback function executes and processes the result.

This example can be extended to multiple worker threads with different operations by creating more worker instances with varying locations of script given for the source parameter in the worker thread constructor. Similarly, any CPU-intensive tasks can be distributed among different worker threads, ensuring that the main thread's event loop is not blocked.

2. Scaling out to multiple servers/clusters

When an application faces a spike in demand, horizontal scaling can become handy. However, when it comes to Node.js applications, they can be scaled using two techniques.

Scale with clustering.
Scale across multiple servers.

Scale with clustering

Clustering is commonly used to scale a Node.js application horizontally within the same server. It allows developers to take full advantage of a multi-core system while reducing application downtime and outages by distributing the requests among the child processes.

You can create child processes that run concurrently with the application sharing the same port, which helps scale the application within the same server.
Clusters can be implemented using the built-in cluster module of Node.js or a library like PM2, widely used in production applications.

const cluster = require('cluster');
const http = require('http');
const numCPUs = require('os').cpus().length;

if (cluster.isMaster) {
  console.log(`Master ${process.pid} is running`);

  // Fork workers.
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  cluster.on('exit,' (worker, code, signal) => {
    console.log(`worker ${worker.process.pid} died`);
  });
} else {
  // Workers can share any TCP connection
  //, In this case,, it is an HTTP server
  HTTP.createServer((req, res) => {
    res.writeHead(200);
    res.end('Hello World\n');
  }).listen(8000);

  console.log(`Worker ${process.pid} started`);
}

The snippet above highlights the use of horizontal scaling via clustering. It uses the built-in cluster module to fork and create child processes based on the available CPU count.

As shown above, you must write a decent amount of code to handle clustering, even for a simple application. Unfortunately, this approach is not maintainable for complex mission-critical production applications.

Therefore, libraries like PM2 can come in handy to handle all the added complexity behind the scenes. All we need to do is add PM2 on the server globally and run the application with PM2, which will spawn as many processes as possible in the system.

Additionally, it provides many more features to manage and visualize the processes. More details on PM2 can be found in their official documentation.

Scale with multiple servers

A Node.js application can be scaled horizontally using multiple servers as long as the application is running as an independent process. A load balancer can be introduced to handle scaling across servers, where the load balancer will distribute the requests among servers depending on the load.

However, using a single load balance is not a good practice as this creates a single point of failure. Therefore, it is best to introduce multiple load balancers pointing to the same server, depending on the application's criticality.

3. Breaking the application into microservices

Microservices is a software architecture pattern that breaks the application into smaller, independent, functional units where each unit can function and scale independently without affecting other services. Additionally, it helps improve scalability in several ways.

Scalability at the component level

Since each service is independent of other services, they can be scaled independently, which reduces the complexity of scaling the whole application. Furthermore, it allows the developers to scale each microservice according to the client's requirements. For example, you can select and only scale the microservices with high traffic.

Improved reliability

Breaking down a large monolith application into smaller, independent microservices makes observing and tracing each service easier. This improved observability can help identify and isolate issues within the system, making it easier to implement failover strategies that minimize the impact on the overall application.

Additionally, because each microservice operates independently, a failure in one service will only affect that specific service and not the entire application.

Efficient scalability

The small size of the services allows better resource utilization where needed. Hence, when the demand increases, the application can effectively scale while utilizing the available resources.

Additionally, microservices help contribute to the overall efficiency of development time. It can also result in faster deployments and releases. Moreover, developers can also utilize third-party tools to simplify microservice generation.

Tools like Amplication help developers generate fully functional services based on Typescript and Node.js with popular technologies such as (but not limited to) NestJS, Prisma, PostgreSQL, GraphQL, and MongoDB while ensuring your services are highly scalable and secure.

4. Optimizing static assets through a CDN

Node.js is fast when it comes to handling dynamic content like JSON objects. However, Node.js tends to underperform when you try to manage static assets such as images. Also, serving static content from the Node.js applications will be resource-intensive and could increase the application's latency.

To avoid this, you can use services like Nginx or Apache to serve static content. In addition, these web servers can also optimize serving static content and cache it on the web server.

Secondly, you can use specific services like Content Delivery Networks (CDNs) that are built to serve static content. CDNs have edge (POP) locations worldwide, bringing the static content closer to the users, thus improving the latency significantly while freeing the Node.js application to handle the dynamic requests.

We'll give it a shot!

Whoa, we're half way there... with this article. So, if you're pumped up after reading this article so far, let's keep the energy going and show some love for our project!

We're living on a prayer, working hard to build amazing tools for the Node.js community. But we can't do it alone! We need your support to keep pushing the boundaries and creating extraordinary experiences for developers like you.

Give us a shout-out and a 🌟 to the Amplication repository on GitHub.

🤘

5. Stateless authentication

Stateless authentication is an authentication technique in which most session details, such as user properties, are saved on the client side, while the server has no information on any previous requests.

It is generally implemented using token-based authentication via a JWT (JSON Web Token), which contains information (basic information + authorities) about the logged-in user. The JWT token is created and validated using a private and public key combination.

The client will send the JWT token to the server on every request made to the server, where the server validates the token to authenticate and verify the request. This lets the Node.js backend scale without any dependencies on the user sessions, as the server has to match the signature against the hash of the payload and the header generated using the private key.

This process reduces the workload on the server side to validate user requests and makes the authentication process scalable, as it has no dependencies on any specific server.

6. Use of timeouts for I/O operations

The performance of an application can be affected by external services despite the app's resilience and high performance. Therefore, it is vital to implement timeouts to ensure that the application is not waiting for a response from a different service for an extended time. To do so, developers can utilize the built-in timeout operations in the third-party services.

const axios = require('axios');

// set timeout to 5 seconds; after no response for 5 seconds, the request is timed out to process request faster.
const instance = axios.create({
  timeout: 5000
});

instance.get('https://test.com')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    if (error.code === 'ECONNABORTED') {
      console.error('Timeout occurred');
    } else {
      console.error(error);
    }
  });

The snippet above depicts an example of a timeout.

It ensures that all API requests that execute through Axios will automatically be timed-out if it doesn't receive a response from the service after 5 seconds. It is implemented using the timeout attribute of the library and avoids blocking subsequent requests.

7. Implement tracking, monitoring & observability to debug and solve performance issues actively

Identifying performance bottlenecks of applications is crucial when building a scalable application. Therefore, implementing tracing, monitoring, and observability can provide insights into the actual bottlenecks and can help resolve the issues quickly.

Tracing is the ability to track a request in each stage. It helps to trace the exact sequence of events in a request before it finishes its execution. This provides developers with the information they need to determine areas of concern regarding latency. They can inspect and fix parts of the request before it becomes a significant issue.

Monitoring refers to tracking fundamental metrics of the application, such as response time, error rates, and resource utilization so that developers are kept aware of all activities within the application. They can then utilize the generated logs to troubleshoot errors within the application while using the metrics to identify areas of bottlenecks.

Finally, developers can combine tracing + monitoring and gain a holistic standpoint of the application through observability. Observability helps developers determine the internal state based on the generated outputs (metrics + logs). Hence, developers can easily find and isolate performance issues and fix them quickly.

Many open-source and commercial tools are available to implement tracing and monitoring in applications to improve the application's observability. These tools provide extensive insights into the application's performance and can help identify issues and improve the application's performance.

Conclusion

In this article, we discussed 7 tips for building scalable Node.js applications. Adopting these techniques will improve the performance and scalability of your Node.js application.

However, it is essential to identify and isolate bottlenecks before jumping in with a solution. Also, it's critical to understand that bottlenecks could change as your application scales. Thus, you may notice hot spots in different application parts at different scales. So, it is vital to maintain observability and regularly monitor the critical factors to identify the issues early.

How Amplication can help

Amplication is an open-source platform that helps you build backend services without spending time on repetitive coding tasks and boilerplate code. Instead, Amplication auto-generates a fully functional, production-ready backend based on TypeScript and Node.js.

Whether you build a single service or a microservices architecture, Amplication allows you to build at any scale.

With Amplication, development teams can create multiple services, manage microservices communication, use Kafka, connect to storage, or add an API Gateway.

Amplication can sync the generated code with a monorepo where each service goes to a different folder or with various repositories. You can manage dozens or hundreds of services with maximum consistency from a single source of truth and centralized management and visibility.

Monoliths to Microservices using the Strangler Pattern

Yuval Hazaz — Thu, 11 May 2023 09:17:05 +0000

Microservices architecture has many advantages compared to monolithic architecture. Hence, many organizations start building new applications with microservices or convert existing monoliths to microservices.

However, transforming a large-scale monolithic application to microservices from scratch is a challenging task. It requires significant architecture and development effort and poses a higher risk to the organization. Nevertheless, some well-established patterns and practices may help to reduce these risks.

Let's look at how we can use the Strangler pattern to transform a monolith into microservices.

Why do we need the Strangler Pattern?

According to Statista, over 80% of organizations worldwide use microservices, and 17.5% plan migrating their existing architectures to microservices. But, writing an existing monolithic application from scratch with microservices is a challenging task. As mentioned, it involves severe risks and requires significant developer effort.

For example, here are some of the challenges you might face when writing an application from scratch:

Time-consuming - Rewriting an application can take a significant time. Especially if your application is large-scale, it can take up to years. Also, there can be unexpected issues that can take a considerable amount of time to resolve.
Can't develop new features - The rewriting process requires a colossal developer effort. You might have to divert the whole development team to the migration task, which will slow down the new feature development.
Uncertainty - You must wait to use the new system until the development process completes. Then only you can ensure that all the existing functionalities are working as expected.

The Strangler Pattern minimizes these risks and uncertainties while providing a way to gradually refactor monolithic applications to microservices rather than writing from scratch.

Furthermore, organizations don't have to pause new feature development or wait long to see a working application. Instead, they can choose components based on priority and refactor them to microservices while keeping monolith and microservices as a single application.

Now we are beginning to understand the need for the Strangler Pattern. So, let's get into more detail on the Strangler Pattern and how it works.

What is the Strangler Pattern?

Strangler Pattern is a software design pattern used to refactor monolithic applications to microservices gradually. It helps developers to replace parts of the monolith with new and improved components while maintaining the same functionality.

The Strangler Pattern uses a wrapper to integrate the microservices with the monolith. This wrapper is an integral part of this design pattern since it bridges the monolith and the microservices, directing incoming requests to the appropriate component for processing. Furthermore, it acts as a fail-safe, allowing the organization to roll back to the monolith if there are any issues with the new microservice.

To get a better understanding, let's discuss how Strangler Patterns work using a real-world scenario.

A quick word from our sponsor

Are you enjoying this article about the Strangler Pattern and how you can migrate from a monolith to microservices? If so, we invite you to check out Amplication, the low-code development platform that can help you make that migration with ease. And, if you appreciate our mission to simplify app development for everyone, please consider showing your support by giving our repo on GitHub a 🌟. In return we'll send you an imaginary star which you can stick on anything! Thank you for your support.

How does a Strangler Pattern work?

Refactoring a monolith into microservices with the Strangler Pattern consists of 3 main steps: Transform, Coexist, and Eliminate.

Transform: You need to start by identifying the main components of the monolithic application. This step involves identifying the boundaries between the existing application and the new components being developed.
Coexist: Then, build a wrapper around the monolith to allow the new components to coexist with the existing application.
Eliminate: Finally, eliminate the monolith by replacing parts with new components. However, you must ensure that each microservice works as expected before integrating it into the system.

For example, consider an e-commerce application with a monolithic architecture. The application will include user registration, product search, shopping cart, payment handling, inventory management, and more.

To give you a better understanding, I will divide the migration process into five steps.

Step 1 - Deciding the first microservice

As the first step, you need to individually identify the monolith components with their capabilities and limitations.

Then, it would be best to decide on the first microservice you will migrate. There are several factors to consider when ordering the components for refactoring, and I have explained them in detail in the next section.

In this example, I have decided to use product search functionality as the first component to migrate since it is not in the application's critical path.

Also, it is crucial to identify the dependencies between the selected component and others. In this example, the product search component is directly connected with inventory and shipping cart components.

Step 2 - Creating the new microservice

As the second step, you must start creating the new microservice and move all the relevant business logic to the new microservice. Then, it would help if you created an API for the microservice, which will act as the wrapper to handle communications between the monolith and the microservice.

After implementing the microservice, you need to run both the monolith's component and the microservice in parallel to verify the functionality.

Step 3 - Handling the databases

When you create a new microservice, it is necessary to create a separate database. In addition, since we are maintaining both monolith and microservice in parallel for a time, we also need to keep the primary and microservices databases in sync.

For that, we can use a technique like read-through-write-through caching. In this example, we will use the microservice database as a cache until we remove the product search component from the monolith. Then, when a user searches for a product, we look in the microservices' database. We can fetch the data from the primary database if it is not there.

Step 4 - Handling requests

Initially, routing a small percentage of traffic to the new search service is advised to reduce the blast radius of the new microservice. For that, you can use a load balancer.

Step 5 - Test and repeat

Once you verify the functionality of the new microservice, you can remove the component in the monolith and route all the traffic to the microservice. Then, you must repeat these steps for each component and gradually convert the monolith into microservices.

How to select the component order for refactoring?

Another major issue developers face when using the Strangler Pattern is the component order for refactoring. Although no hard rules are defined for the selection process, it is essential to sort out what components should be migrated first to avoid unexpected delays.

Here are some factors you need to consider when deciding the order of the components:

Consider dependencies: It is good to start with components with few dependencies, as these are likely to be easier to refactor.
Start with low-risk components: Starting with low-risk components will minimize the impact on the system if something goes wrong in the early stages. Also, it will help you to gain experience and build confidence in the process.
Business needs: If there is a high-demand component and you need to scale it as soon as possible, you should start with that component to facilitate the business requirement.
Components with frequent changes: If components require frequent updates and deployments, refactoring them as microservices will allow you to manage separate deployment pipelines for those services.
User experience: Start with the components with the most negligible impact on end-users. It reduces disruptions and helps to maintain a good user experience during the transition.
Integrations: Refactor components with minimal integration with other systems first.

Apart from the above, there are many other factors you can consider. However, ultimately you need to consider the most prominent factors for your project and order the components for the refactoring process.

Different ways to implement the Strangler Pattern

Unlike other design patterns, the Strangler Pattern has no language-specific libraries to help developers implement it. Instead, developers must use technologies, frameworks, and best practices to implement the Strangler Pattern.

Here are some of the most used approaches in implementing the Strangler Pattern:

Using ready-made platforms: Instead of building the infrastructure and platform architecture for microservices from the ground up, you can consider utilizing ready-made platforms that mostly do the heavy lifting. E.g., Amplication, Strapi, AppWrite
Using serverless: You can use AWS Lambda or Google Cloud Functions to implement independent functions triggered by specific events and eventually replace the parts of the monolith with them.
API gateways: An API gateway can be the wrapper when implementing the Strangler Pattern. It provides a unified interface and can be configured to redirect requests to the appropriate component. Amazon API Gateway, Kong, and Tyk are some popular API gateways you can use.
Reverse proxies: A reverse proxy like Nginx can also be used as the wrapper in the Strangler Pattern.
Routing and load balancing: Routing and load balancing technologies can redirect traffic to the appropriate components. DNS-based routing and software-defined load balancers are popular options you can use.
Service discovery: You can get the help of the service discovery pattern to find the locations of new microservices.
service mesh: You can use technologies like Istio or Linkerd to manage the communication between the new components.

These are only a subset of tools and technologies you can use to implement the Strangler Pattern. But make sure to only select a limited number of technologies from them based on your requirement. Otherwise, you will end up over-engineering the system.

Advantages of the Strangler Pattern

Incremental migration: Allows for an incremental migration from a monolith to microservices and reduces the risk associated with the migration process.
Reduced downtime: Ensures the system remains operational throughout the migration.
Improved resilience: Improve the system's resilience by ensuring that the monolith and microservices coexist and work together seamlessly.
Increased flexibility: Allow the organization to choose the best technology for each part of the system, rather than being forced to use a single technology for the entire system.
Better maintainability: Breaking down the monolith into microservices makes it easier to maintain the system over time.

Challenges of the Strangler Pattern

Modularizing complexity: Breaking a monolith into components is difficult when the functionalities are tightly coupled.
Data compatibility: Sometimes, you may need to perform a data migration or transformation when the monolith and the microservices use different data formats.
Testing effort: Extensive testing is required to ensure the new microservices work as expected.
Skill requirements: Using the Strangler Pattern requires a high level of technical skill. Implementing the pattern in organizations that need more technical expertise can make it challenging.

Speedup the microservice generation

This article discussed how the Strangler Pattern simplifies the conversion of monoliths to microservices while highlighting the advantages and challenges of the process.

However, converting a monolith with hundreds of components is challenging, even with the Strangler Pattern. You need to design the underlying architecture for microservices and work on creating each service from scratch.

One solution is to create a blueprint for each microservice and develop the tools to generate it. You can even consider developing a domain-driven language to standardize and reuse best practices in code and configuration. However, it also adds complexity and cost to your architecture, where you need to manage and evolve these tools in the long run.

How Amplication can help

Amplication can manage all the underlying complexities while simplifying the creation of microservices.

Amplication supports microservices through the project hierarchy. A project groups together multiple resources used and created by Amplication, enabling support for various use cases. This simplifies the creation of connected services and makes syncing with GitHub across multiple Services much easier.

You can find a getting started guide here.

ORM, Prisma, and How You Should Build Your Next Backend Database Project

Yuval Hazaz — Tue, 25 Apr 2023 06:44:23 +0000

Prisma is a premier object-relational mapping (ORM) tool in the Node.js ecosystem. It's powerful yet simple and easy to use. A great team also supports it, and it has enjoyed widespread adoption by the Node.js community. Prisma has earned such an excellent reputation that we incorporated it into our tech stack for Amplication. Not only do we use Prisma for our generated applications, but we also use it in our internal tech stack.

In this article, we'll closely examine some of Prisma's features and the team that supports it. Prisma has become as crucial to Amplication as Node itself, and I highly recommend you consider it for your next Node project.

Introducing Prisma

Prisma is one of the most popular ORMs in the node ecosystem, and over the last two years, it has seen consistent growth in the community. Much of this has to do with the team supporting the product. Prisma is open-source and licensed under Apache-2.0, making it very flexible for your projects—an impressive array of investors, such as key members from Heroku, Netlify, and GraphQL, back it. Support for Prisma is ongoing and robust, as the team continuously makes improvements with regular releases that address bugs, performance issues, and other enhancements.

One of the most important considerations when choosing a tech stack is ensuring that any application or package I use is well supported. No matter how performant or sophisticated a package is, if it's under-supported, it can add untold hours to your project as you try to fit it into an edge use case or work out a bug. Prisma's popularity and open-source status mean that not only is it supported by a fantastic internal team, but the community support is incredible, too. There are plenty of well-educated and experienced Prisma developers on the usual forums.

Prisma is a package that invites new users. It is well-documented and well-suited for your first or 50th Node project. Their documentation contains comprehensive information on the features available. You can also access several guides, including introductory "how-tos," steps for deploying applications that use Prisma, and instructions on migrating from another ORM. The guides are thorough, up-to-date, and easy to follow. In addition, Prisma's team has made sure your experience using their ORM is as painless as possible.

Why Use Prisma?

In the simplest case, Prisma can access your database as an ORM. As part of its suite of products, Prisma offers a "Client API" that can make writing even the most complex database operations simple. But where Prisma shines is in its ability to handle complex querying operations.

Prisma's API lets you easily traverse relationships. Below is an example of an application accessing a database from the Prisma Client tutorial. First, the application accesses an author's profile by using the navigation properties from the blog post to the author and finally to the author's profile:

const authorProfile: Profile | null = await prisma.post
  .findUnique({ where: { id: 1 } })
  .author()
  .profile();

It also makes pagination a breeze by exposing arguments for order, limits, and cursors. Below you can see an example where you can use the client to take five posts from the database by starting from the post with id=2:

// Find the next 5 posts after post id 2
const paginatedPosts3: Post[] = await prisma.post.findMany({
  take: 5,
  cursor: {
    id: 2,
  },
});

It also allows for aggregate queries such as sum and count:

// Group users by country
const groupUsers = await prisma.User.groupBy({
  by: ["country"],
  _count: {
    id: true,
  },
});

Along with these features, Prisma's client also facilitates transactions, includes middleware and the execution of raw questions, and helps make logging simple.

But to limit Prisma's capabilities to just reading or writing data would be a major disservice. Another great aspect of Prisma is how it handles migrations. Prisma supports SDL-first and code-first approaches to modeling your data structure in code. I suggest the code-first approach to writing your database if creating a new application. Still, creating your Prisma schema using the database for existing databases can be easier.

If you decide to work with the code-first approach, you can use Prisma Migrate to create the tables in your database. Start by writing your schema definition using Prisma's markup language. The following is a sample from the Prisma tutorial:

model Post {
  id        Int      @id @default(autoincrement())
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
  title     String   @db.VarChar(255)
  content   String?
  published Boolean  @default(false)
  author    User     @relation(fields: [authorId], references: [id])
  authorId  Int
}

model Profile {
  id     Int     @id @default(autoincrement())
  bio    String?
  user   User    @relation(fields: [userId], references: [id])
  userId Int     @unique
}

model User {
  id      Int      @id @default(autoincrement())
  email   String   @unique
  name    String?
  posts   Post[]
  profile profile?
}

With your schema set up, you can prepare your database by selecting one of the providers supported by Prisma. In the example below, I'm using PostgreSQL, but there are several available options like MySQL, SQLite, MongoDB, CockroachDB, and Microsoft SQL Server:

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

With your schema and provider set, Prisma is now able to convert your schema into executable code to create your database:

-- CreateTable
CREATE TABLE "User" (
  "id" SERIAL,
  "name" VARCHAR(200) NOT NULL,
  PRIMARY KEY ("id")
);

-- CreateTable
CREATE TABLE "Post" (
  "id" SERIAL,
  "title" VARCHAR(150) NOT NULL,
  "published" BOOLEAN NOT NULL DEFAULT true,
  "authorId" INTEGER NOT NULL,
  PRIMARY KEY ("id")
);

-- AddForeignKey
ALTER TABLE "Post" ADD FOREIGN KEY("authorId")REFERENCES "User"("id") ON DELETE CASCADE ON UPDATE CASCADE;

It's just that simple. If you switch from PostgreSQL to MySQL (or any other provider), change your provider and rebuild your migration. If you need to create seed data, you can configure your application to know where your seed data is and use the Prisma client to insert any data you need.

Using Prisma With Amplication

Of course, you need to be able to get the data from your database to your User. Amplication will create a schema.prisma file from the configuration you inputted into the UI. This file will contain all the fields, types, and relationships necessary to create your databases. Each entity you make in the UI will also generate a series of classes. Since Amplication is built on NestJS, we generate a module, a resolver, and a service to facilitate handling queries to the API for the entity.

First, the service is where we use Prisma to query or modify data in the database. We generated typed classes to represent the arguments for each query, such as CustomerFindManyArguments. Then we create wrapper methods for Prisma such as findMany, which calls the customer collection's findMany function from the Prisma client and passes the instance of CustomerFinderManyArguments to it. The Prisma client uses these arguments to filter your data or use pagination.

Amplication will then generate each entity's REST API controllers and GraphQL resolvers. First, we generate the typed classes; we use the same entity definition you provided to build prisma.schema. This helps sync the API and database regarding required fields and types. Then we can use those typed classes to inform Swagger of the structure of the application's API. Finally, we register the modules to the application and then the application to Swagger. You'll notice that the fields in your Args classes are decorated with APIProperty. This gives Swagger all the details it needs to create your API documentation.

A closer look at the CustomerFinderManyArguments class reveals that we also use decorators from the @nestjs/graphql. By doing this, we can provide GraphQL of the available fields in your data. We then generate a resolver to select what the application will expose via GraphQL. The resolvers depend on an entity service generated with your application.

Finally, Amplication will generate Swagger documentation for your API that fully represents your end-to-end data transfer.

You can review the CustomerServiceBase in our sample application for a closer look.

Figure 1: Swagger UI (Source: Amplication)

Wrapping Up

Now you can see why Prisma has become integral to the Amplication stack. It is simple and easy to use but powerful and fully featured. The team supporting the product has done a fantastic job of enhancing it, too. As a result, Prisma will be a part of our stack for the foreseeable future, and we recommend you make it a part of yours as well.

If you're excited about Amplication and its potential to simplify your development workflow, please consider showing your support by starring our repository on GitHub. Your 🌟 will help us reach more developers and continue to improve the platform.

SQL Versus NoSQL Databases: Which to Use, When, and Why

Yuval Hazaz — Fri, 21 Apr 2023 08:14:29 +0000

Choosing a suitable database for storing specific data types is vital in software development. The right choice means we can scale up an application quickly and handle increasing user requests without encountering problems. Appropriate data storage also gives us an easier time when adding new features to the application. It lets us manipulate users' data effectively and efficiently. However, choosing the correct database for your application is no mean feat, given the number of available databases.

Databases can be categorized into two primary types: SQL, which is an acronym for Structured Query Language, and NoSQL, which stands for "not only SQL" or "non-SQL." In this article, we'll review the differences between SQL and NoSQL databases, examine their appropriate use cases, and look at how to work with each using Node.js, so you can confidently opt for the right one in your next project.

Differences Between SQL and NoSQL Databases

There are many differences between SQL and NoSQL databases, such as how they handle relational data, their query languages, and their supported tools for development purposes. In this section, we'll look at their most significant differences.

Handling Relations

SQL databases handle data relations using tables. The tables can be interconnected using key constraints. There are three types of relations in SQL databases: one-to-one, one-to-many, and many-to-many.

On the other hand, NoSQL databases do not support handling data relations. So, for example, if we want to combine data from two documents in a MongoDB database, we need to write our own code to apply the logic we want.

Schema vs Schemaless

Schema is a term that describes the objects in a database, such as tables, views, relationships, and indexes. All SQL databases are schema databases. However, different SQL databases treat their schemas differently from one another. In Oracle and MySQL, there is no specific database object named schema; however, they have other database objects—like tables, views, and relationships. PostgreSQL and SQL Server have a specific database object: a schema. An SQL database can also have multiple schemas, as in the case of PostgreSQL.

By contrast, NoSQL databases are schemaless. As such, NoSQL databases provide flexibility when storing user data. We don't need to keep the data structurally consistent, as in SQL databases. Taking MongoDB as an example, we can store user data in different formats so long as it is in BSON (the data type that MongoDB supports.)

Querying Data

In SQL databases, the querying languages are mostly the same. There are some differences in the syntax and how the databases execute actions based on each syntax, but they're just minor cases. For example, with PostgreSQL, let's say we want to query all the users with all columns from the user table. We can write your query like this:

select * from users;

To list all databases in Postgresql, we write:

\l

With another SQL database, such as MySQL, the syntax is different when we try to list all the databases:

show databases;

In NoSQL databases, the query language varies much more. Each of the NoSQL databases has its query language. For example, in MongoDB, if we want to search for all users from the users' collection, we write the query as:

db.users.find({})

Whereas with another NoSQL database, like Redis, if we want to retrieve the value of a key in a Redis database, we write the query as:

GET {key_name}

Scalability

There are two ways to scale the database for SQL and NoSQL databases: horizontal and vertical scaling. With horizontal scaling, we add more nodes to the databases. With vertical scaling, we add more RAM and CPU to the database node.

Horizontal Scaling for SQL Databases

Horizontal scaling for SQL databases is typically handled differently than for NoSQL databases. With SQL databases, like PostgreSQL, we can apply sharding or add replicas to scale the database horizontally.

If our use case targets the read capability, adding replicas is a great option; PostgreSQL provides built-in functionality for scaling replicas to improve readability. However, if we are targeting both read and write capabilities, the solution is much more complicated. PostgreSQL is not designed with heavy write capability or support for distributed databases. We must apply sharding data on partitioning tables or use logical replication to achieve both write and read capabilities.

Horizontal Scaling for NoSQL Databases

With NoSQL databases—say, MongoDB—we can use sharding and replica sets for horizontal scaling. Sharding is usually a preferred option for replica sets when performing horizontal scaling in MongoDB. With sharding, we copy the slices of the data from the primary node to multiple replica sets, and then the replica sets work together to join pieces of the data into a complete dataset. With replica sets, we copy the whole data from the primary node to other nodes. As a result, the write data capacity using replica sets is lowered than sharding.

The differences between sharding and replica sets for horizontal scaling in MongoDB are complex and worth further exploration.

Node.js Tooling

In the Node.js environment, several tools offer support for SQL or NoSQL databases or, in some cases, for both. Prisma is one of the libraries that supports working for SQL and NoSQL databases. With Prisma, we can work with several databases such as PostgreSQL, MySQL, MongoDB, or SQL Server.

SQL: MySQL

Let's say our SQL database has two tables: Author and Blog. The table Blog and Author can join using the authorId key. We will define these two tables in Prisma using schema.prisma file as below:

//schema.prisma
model Blog {
  id Int @id @default(autoincrement())
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
  title String @db.VarChar(255)
  content String?
  published Boolean @default(false)
  author Author @relation(fields: [authorId], references: [id])
  authorId Int
}

model Author {
  id Int @id @default(autoincrement())
  email String @unique
  name String?
  blog Blog[]
}

In the Blog model, we defined an author field related to the model Author by authorId.

To migrate the currently defined schemas with our MySQL database, we run the following:

npx prisma migrate dev --name init

(Note that this command is used for development purposes only. Production database migration requires a different process.)

We should see similar output from the console as the one below:

The following migration(s) have been created and applied from new schema changes:

migrations/
  └─ 20230212011801_init/
    └─ migration.sql

Our database is now in sync with our schema. To add several authors to the Author table, we need to write the following code:

await prisma.author.createMany({
  data: [{
      email: "donald.le@gmail.com",
      name: "Donald Le"
    },
    {
      email: "evans.chris@gmail.com",
      name: "Evan Chris"
    }],
});

Checking the Author table in the database, we see these two authors were added.

+----+-----------------------+------------+
| id | email | name |
+----+-----------------------+------------+
| 1 | donald.le@gmail.com | Donald Le |
| 2 | evans.chris@gmail.com | Evan Chris |
+----+-----------------------+------------+

To add these authors' blog posts to the database, run the following:

await prisma.blog.createMany({
  data: [{
      title: "A great blog",
      content: "A great blog",
      authorId: 1
    }, {
      title: "Another great blog",
      content: "Another great blog",
      authorId: 2
    }],
});

Checking the Blog table, we see these two blogs were added:

+----+-------------------------+-------------------------+--------------------+--------------------+-----------+----------+
| id | createdAt | updatedAt | title | content | published | authorId |
+----+-------------------------+-------------------------+--------------------+--------------------+-----------+----------+
| 1 | 2023-02-12 01:30:35.629 | 2023-02-12 01:30:35.629 | A great blog | A great blog | 0 | 1 |
| 2 | 2023-02-12 01:30:35.629 | 2023-02-12 01:30:35.629 | Another great blog | Another great blog | 0 | 2 |
+----+-------------------------+-------------------------+--------------------+--------------------+-----------+----------+

To query all blogs of the author with authorId of 1, we write our code as:

async function main() {
  const results = await prisma.blog.findMany({
    where:{authorId:1}
  });
  console.log(results);
}

Here's what we see as output:

[
  {
    id: 1,
    createdAt: "2023-02-12T01:30:35.629Z",
    updatedAt: "2023-02-12T01:30:35.629Z",
    title: "A great blog",
    content: "A great blog",
    published: false,
    authorId: 1
  }
]

NoSQL: MongoDB

Let's say we want to work with a NoSQL database like MongoDB (Mongo Atlas), and we have two topics: Blog and Author. These topics have a relation to each other using authorId. We then define these topics in schema.prisma file as:

//schema.prisma
model Blog {
  id String @id @default(auto()) @map("_id") @db.ObjectId
  title String
  content String
  author Author @relation(fields: [authorId], references: [id])
  authorId String @db.ObjectId
}

model Author {
  id String @id @default(auto()) @map("_id") @db.ObjectId
  email String @unique
  name String?
  address Address?
  blogs Blog[]
}

To apply these defined models to MongoDB, we run the following command:

npx prisma generate

Upon checking the Mongo Atlas database, we see two collections are created.

Figure 1: Two collections created in Mongo Atlas

We can add a new author to the Author collection and also add a new blog for that author with the following code:

await prisma.author.create({
  data: {
    name: 'Donald Mathew',
    email: 'donald.matthew@gmail.com',
    blogs: {
      create: {
        title: 'This is an interesting blog post',
        content: 'Lots of really interesting stuff',
      },
    },
  },
});

Under the Author collection in Mongo Atlas, we can see that the new author has been added successfully.

Figure 2: A new author is added to the Author collection

And under the Blog collection, we see the new blog, too.

Figure 3: A new blog is added to the Blog collection

We can also query all the blogs that belong to the author Donald Mathew with authorID of 63e8580b1076c768fd1fe772 by running the following code:

const allUsers = await prisma.blog.findMany({
  where:{authorId:"63e8580b1076c768fd1fe772"}
});

console.dir(allUsers, { depth: null })

Running the code will show the blogs that belong to the author as:

[
  {
    id: "63e8580b1076c768fd1fe773",
    title: "This is an interesting blog post",
    content: "Lots of really interesting stuff",
    authorId: "63e8580b1076c768fd1fe772"
  }
]

And that's how we can use a Node.js library like Prisma to work with NoSQL and SQL databases. Prisma's documentation page gives further details about this process.

Choosing the Right Database for a Specific Use Case

Now that we've reviewed the differences between SQL vs. NoSQL databases let's look at some specific use cases and determine the database type that would be the most appropriate fit in each instance.

Relational Data

Relational data might include users' personal information, including their full name, date of birth, children, and siblings—the information that would logically be stored in a table. An SQL database would be a good choice for these cases because features like join and primary key help us quickly query the relational data we need. Although a NoSQL database gives us flexibility when storing data, and we can quickly scale the MongoDB database, it is not designed for dealing with relational data.

Flexible Data Structures

With data that has flexible data structures, such as user-generated data, a NoSQL database would be a better choice. For example, if we wanted to research how users interact with our application and which functionalities users use most often to achieve this. In that case, we should first store all user logs in a database and then deep dive into these logs. Again, given the flexibility of the data structures from multiple functionalities in our application, a NoSQL database like MongoDB would be a good fit.

Low Latency Applications

When working with applications that require a very low latency—say, ten or twenty milliseconds, as might be expected in a trading application or an online role-play game—a NoSQL database like Redis would best serve the use case. Redis is known for its speed of reading data. However, some parts of the application that do not require such a fast server response would still benefit from using a traditional SQL database, like storing user profiles, a case in which the benefit of an SQL database's support for relational data outweighs the speed benefits of NoSQL.

Applications Targeting Fraud Detection or Personalization

For organizations and their applications that are designed to detect fraud, like International Consortium of Investigative Journalists, or try to improve customer experience via personalization, as in the case of Tourism Media, a NoSQL graph database like Neo4j is a good match. In these kinds of use cases, the quantity of data we're dealing with is enormous, and the pattern we're searching for in the data is often complex.

Neo4j addresses both of these problems effectively. Its design as a graph database means that Neo4j has a query language that provides a nested queries mechanism, which is much simpler than using join in an SQL database. This helps us to find a pattern in our vast ocean of data with relative ease. Neo4j also offers excellent performance with lightspeed throughput when working with massive quantities of data. This makes Neo4j ideal for fraud detection and personalization applications.

Tired of Writing Boilerplate Code when Working with Databases?

Amplication is an application development platform that generates all necessary boilerplate code, leaving you free to focus on creating business value. In addition, Amplication has first-class support for SQL and NoSQL databases, so you can generate applications quickly using whichever is the best fit for your use case and start shipping code immediately.

REST vs. gRPC - What’s the Difference?

Yuval Hazaz — Wed, 19 Apr 2023 16:38:14 +0000

Most modern applications rely on APIs for clients to interact with them. This dependency makes it critical to design APIs that are efficient, scalable, and uniform in nature. Different frameworks have come into the picture to introduce structure and uniformity into API design.

REST has been around for a long time and is an industry standard for developing and designing APIs. gRPC is a more recent framework introduced by Google to create fast and scalable APIs.

In this article, we'll talk in detail about these frameworks and which one may be better for your use case.

What Is REST?

REST stands for representational state transfer and is the most common architectural style used to design APIs and web-based and microservices-based applications. REST is built on the standard HTTP protocol, where each RESTful web service represents a resource. These resources can be fetched or manipulated via a common interface using HTTP standard methods—GET, POST, PUT, DELETE.

REST APIs are:

Client-server independent: User interface (client) concerns are separated from data storage (server) concerns.
Stateless: communication between the client and server contains all information necessary to process the request.
Cacheable: REST resources are cacheable on the client or server side for improved performance.

REST APIs are designed to provide a uniform interface where all components follow the same rules to interact with each other. In addition, the design is based on a layered architecture, so each component can only view the actual layer it's interacting with.

What Is gRPC?

Google Remote Procedure Call, or gRPC, was created on top of the RPC protocol. The RPC protocol is an open-source, cross-platform, high-speed communication protocol built using HTTP 2.0 and is widely used for inter-service communication in distributed applications.

gRPC is an extension of RPC, where a function hosted on a remote server is invoked from another remote server or machine not necessarily hosted on the same server. The HTTP 2.0 protocol allows gRPC to perform bi-directional streaming, provides built-in Transport Layer Security, and lets developers integrate services programmed in various languages.

gRPC uses HTTP as the transport layer so developers can select function calls instead of predefined options. gRPC also provides loose coupling between the server and client because a long-lived connection is created between the server and client, and a new HTTP 2.0 stream is opened for every RPC call.

My Fellow Coders...

My fellow coders, just as we dared to dream of putting a man on the moon, we now have a new frontier to conquer. The Amplication repository is about to hit 10,000-stars on GitHub, and is a powerful tool that has the potential to revolutionize software development. With your support, we can take it to new heights. I call upon all of you to join this noble endeavor by starring the Amplication repository on GitHub. Together, we can achieve greatness and propel the world of software development to new frontiers. Let us not shrink from the challenge, but rather embrace it with unwavering determination. Thank you, and let us boldly go where no developer has gone before!

Ask not what Amplication can do for you - ask what you can do for Amplication!

Comparing REST to gRPC

Working Model

REST is built on the HTTP 1.1 protocol and lets you define APIs based on request-response models. gRPC, on the other hand, is built on the HTTP 2.0 protocol and uses its bi-directional communication feature and the conventional response-request model. This means that in REST APIs, multiple requests from multiple clients are served sequentially, whereas, in gRPC, numerous requests are processed simultaneously, as HTTP 2.0 allows multiplexing.

The working model of REST also needs built-in codes and standard regulations for when you're defining APIs. gRPC follows a model based on a predefined .proto file, including the standard data exchange guidelines that servers and clients need to follow.

Communication Model

REST and gRPC also use different messaging formats to send requests and receive responses.

REST APIs generally implement JSON for receiving and returning messages between the server and the client. Because JSON is text-based and human-readable, it's more efficient and platform-agnostic than other messaging formats.

gRPC uses protocol buffers for serialization and communication, just like REST uses JSON. Protocol Buffers, or Protobuf, are an efficient and highly packed messaging format for serializing data, which results in swift response delivery. In addition, Protobuf is faster when transmitting messages between systems because it performs the marshaling of message packets (packing parameters and a remote function into a binary message packet) before sending it over the network.

Browser Support and Latency

The HTTP 1.1 protocol provides universal browser support, meaning REST is open for all browsers without any prerequisites. The HTTP 2.0 protocol, on the other hand, provides limited browser support, meaning gRPC is not compatible with many browsers, usually older versions. Below is a list of some popular browsers along with their versions that support the HTTP 2.0 protocol and hence gRPC as well:

Browser	Unsupported Versions	Supported Versions
Google Chrome	4 to 40	41 to 115
Mozilla Firefox	2 to 35	36 to 114
Internet Explorer	6 to 10	11 (partially supports)
Safari	3.1 to 8	9 to 10.1 (partially supports)11 to 16.4 (fully supports)
Microsoft Edge	All other older versions	12 to 112
Opera	10 to 27	28 to 95

Data Formats and Serialization

REST uses multiple data formats like JSON, XML, etc. JSON is the most commonly used, as it's easy to understand and flexible, but REST generally doesn't require a rigid data structure. This flexibility makes it ideal solely for the transmission of non-critical data. Meanwhile, gRPC only supports the Protobuf message format since this allows data to be transferred more reliably.

As mentioned above, Protocol Buffers compress the data for faster transmission. gRPC's use of these buffers results in agnostic serialization, whereas REST offers serialization by converting data into Python-native data supported by XML and JSON.

When to Use gRPC over REST

Before we get to the preferred use cases for gRPC, here are a few words on REST. REST is beneficial when an application needs numerous third-party integrations; it's also suitable for developing cloud-based applications since REST makes stateless calls that can be easily incorporated into cloud applications if any technical failure occurs. REST also supports all kinds of browsers for your application, but this is where we get to the downsides.

In exchange for that excellent browser support, you get high latency with REST. gRPC supports limited browsers, but latency is never an issue. gRPC is the preferred choice when developing lightweight microservice applications because of this reduced latency and faster data transmission. Devs mostly use gRPC with microservices that require real-time message delivery, as it features a very high level of message serialization. It's especially good for connecting systems where lightweight messaging, low-power networks, and high efficiency are required.

Another advantage over REST is that gRPC provides native code generation so developers can develop applications in a multilingual or language-agnostic environment.

All in all, gRPC is used in applications that need multiplexed streams or for mobile applications with limited browser support.

Conclusion

It's tough to say which is better—REST or gRPC. But we can conclude that gRPC is most beneficial for large-scale distributed applications. However, one disadvantage of gRPC is that it's just not popular enough and has limited browser compatibility. This disadvantage means a proxy tool may be needed to process a request from the browser to the gRPC server.

Request Tracing in Node.js

Yuval Hazaz — Thu, 13 Apr 2023 15:27:54 +0000

There’s a saying that software developers like us spend 90% of our time on debugging, and only 10% of our time actually writing code. This is a bit of an exaggeration! It is true that debugging is a significant part of our work, though, especially in this era of microservices architecture. Today, it’s not unusual for us to have hundreds—even thousands—of microservices running simultaneously in our production servers.

Traditionally, we rely on logs when it comes to debugging software problems. However, not all logs are helpful. They can be unspecific, for example, indicating only the error status code or showing a generic error, such as “Something went wrong.” Even if the log records a more specific error—like “User request id is invalid” with a “400” bad request error—it can still take us hours or days to figure out the root cause of the problem due to the high number of services involved. This is where request tracing comes into play.

By applying request tracing in our Node.js application, we can see the flow of the problematic error represented visually. This helps us to pinpoint which services may be involved in the error, and find the root cause of the problem quickly. In this article, we’ll define request tracing, explore its importance, and look at how we can efficiently apply request tracing in our Node.js application.

What Is Request Tracing?

Request tracing refers to the technique of finding subrequests from the multiple services that are triggered when a single request is made. In a microservices architecture, especially in a distributed application, a service often needs to integrate with other services, databases, or third-party dependencies. As a result, when a request is executed, inner requests are triggered too.

Why We Need Request Tracing

In order to debug an error, we need to understand how the request that generated the error is created. Logs alone are not enough; they only tell us which error type has occurred, but do not tell us the context of the issue.

Traces, on the other hand, show us the requests executed inside a single parent request. We can also see the request that was made immediately before the parent's request. In short, request tracing provides us with an overview of the request process. By combining traces with logs, we can identify the root cause of the problem. Moreover, tracing reduces our debugging time since it helps to highlight the true origin of an issue in a simple and quick way by passing log error cascades.

An Amplication Request

Hey developer, while you're reading about request tracing, we have a request for you... We'd love to get a star from you on GitHub for Amplication. Think of it as a virtual high-five that makes our developer's hearts skip a beat. Plus, when we reach 10,000 stars, I've promised the team a pizza party. So, if you want to see some thrilled developers with tomato sauce on their faces, give us a 🌟 on GitHub!

Please don't leave us hanging like a callback without a promise! So click that shiny star button and spread some Amplication love.

How Request Tracing Works

When a user performs an action—for example, clicking the “Confirm” button to buy a shoe from an e-commerce store—several requests are executed. First, a new trace will be created after the user clicks “Confirm.” Then a parent span will be triggered. Inside the parent span, there could be an API request to check the authenticity of the user, and another API to confirm whether the user has permission to make the payment. Additional API requests could be executed, such as checking the number of shoes in stock is sufficient, or verifying that the user balance is high enough to make the payment.

These requests will all be captured and included in the parent span. As a result, we can see how these requests correlate to one another, and if a request fails, we can tell what happened immediately beforehand that may have acted as a trigger.
How to Implement Request Tracing for Node.js Applications.

To implement request tracing for the Node.js application, we need to assign a unique request ID for every request in the current service so that it will forward to other services. As a result, we can use this ID to trace and visualize requests across our system architecture and identify the root cause of the failure.

Common Tools that Support Request Tracing

There are a number of open source observability backend tools—like Jaeger, Zipkin, and Signoz—that support storing and visualizing request traces. Previously, two popular standards existed for the traces format: OpenTracing and OpenCensus. Tools often only supported one or another, making it a challenge to integrate software applications with the tools they need. Fortunately, OpenTracing and OpenCensus unified into OpenTelemetry in 2019, allowing software developers to implement request tracing with relative ease.

OpenTelemetry is a vendor-agnostic collection of tools that can visualize traces, collect data on the time spent processing each request, and capture the requests as individual spans in traces. In order for different observability tools to visualize traces, applications need to follow OpenTelemetry formats.

Implement Request Tracing as a Middleware

Let’s say we have an application that lets users create blog posts to review movies. The application will provide APIs for users to create an account, log in, and create new blog posts.
We will use OpenTelemetry to apply request tracing for the application. The traces then will be forwarded to Jaeger. Our application will use MongoDB as a database and Express as a web server. We need to install the following dependencies in order for OpenTelemetry to attach traces to the API request.

npm install --save @opentelemetry/api
npm install --save @opentelemetry/sdk-trace-node
npm install --save opentelemetry-instrumentation-express
npm install --save @opentelemetry/instrumentation-mongodb
npm install --save @opentelemetry/instrumentation-http

We also need to install the Jaeger dependency to forward OpenTelemetry traces to Jaeger so that we can view the traces via the Jaeger GUI page later.

npm install --save @opentelemetry/exporter-jaeger

To implement request tracing as a middleware, let’s create a new file in the root directory of the project, named tracing.js. Copy the following code to the file:

const { Resource } = require("@opentelemetry/resources");
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions");
const { SimpleSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { trace } = require("@opentelemetry/api");
//exporter
const { JaegerExporter } = require("@opentelemetry/exporter-jaeger");
//instrumentations
const { ExpressInstrumentation } = require("opentelemetry-instrumentation-express");
const { MongoDBInstrumentation } = require("@opentelemetry/instrumentation-mongodb");
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");

//Exporter
module.exports = (serviceName) => {
  const exporter = new JaegerExporter({
    endpoint: "http://localhost:14268/api/traces",
  });

  const provider = new NodeTracerProvider({
    resource: new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
    }),
  });

  provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
  provider.register();
  registerInstrumentations({
    instrumentations: [
      new HttpInstrumentation(),
      new ExpressInstrumentation(),
      new MongoDBInstrumentation(),
    ],
    tracerProvider: provider,
  });

  return trace.getTracer(serviceName);
};

In the code, we’ll register instrumentation for the application to attach traces to MongoDB requests, Express server requests, and HTTP requests:

  instrumentations: [
      new HttpInstrumentation(),
      new ExpressInstrumentation(),
      new MongoDBInstrumentation(),
  ]

The traces will be sent to the Jaeger endpoint: http://localhost:14268/api/traces.
We also need to add one line of code to the entry file of the application in order to enable OpenTelemetry to send traces to the application:

const tracer = require("./tracing")("blog-movie-review-service");

Before we start our application to check for the request traces, we need to bring up the Jaeger service to store the traces. Run the following command to trigger the Jaeger service using Docker:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.32

Let’s bring up the application, and then go to the Jaeger GUI page to see the captured traces.

Figure 1: Captured traces are shown in Jager GUI page

We can see that the traces have been captured by OpenTelemetry and are being stored in Jaeger. To check the trace related to the API request for creating a new blog post, click on the first trace (Figure 1) to bring up details (Figure 2).

Figure 2: Detail about the trace for success request to create a new post

Here, we can see inside the trace, there are spans for HTTP request (POST /v1/posts) and MongoDB request (mongodb.insert).
Let’s see what a failed API request looks like when we provide an invalid authorId for the post:

Figure 3: Detail about the trace for failed request to create a new post

Here, we see that the API request failed to make an HTTP request because the application is verifying the validity of the authorId, and there is no request being sent to MongoDB to insert the blog post.
By applying request tracing to our application, we can easily debug the application and discover at which step in the API request the problem arises.

Best Practices for Request Tracing

As we have seen, applying request tracing for a Node.js application enables us to investigate software issues quickly and effectively. To achieve optimal request tracing results, we should follow three core best practices when applying request tracing for our application.

1. Follow OpenTelemetry Standard Formats

Following OpenTelemetry standard formats allows other tools that can integrate with OpenTelemetry (like Zipkin, Jaeger, or Prometheus) to store and visualize traces effectively. We can choose the tool that suits our specific needs, say, to achieve better visualization without applying a new trace format.

2. Choose a Unique String for Every Source

Choosing a unique string for every source allows us to trace the software issue quickly based on the unique string. Let’s say we want to trace the failed API request for creating a new blog post in the demo application. From the application log, we obtained the unique trace ID used to make the API request. We can grab that unique trace ID and search Jaeger to obtain a full picture of what went wrong.

3. Choose an Appropriate Span Name

Choosing a span name that matches the name of the method invoked by the span allows us to figure out what was going on in the application when the problem occurred. We should also include a custom span tag to preserve the trace. This enables us to search for the custom tag directly, instead of having to locate the trace ID and then search for it.
For example, we can add a custom span tag for the trace related to creating a new blog post. When we need to investigate an issue related to creating blog posts, we can use the custom span tag, and all the traces for creating new movie review posts will be displayed instantly.

Conclusion

In the world of distributed software and microservice architecture, an application log alone is not enough to debug when a problem arises. To achieve observability so that we can debug software problems quickly, we need to implement request tracing in our Node.js application. OpenTelemetry’s standard tracing format makes achieving traceability easy, since we can simply make use of one of the many tools readily available to aid in the request tracing process.