DEV Community: STEVE

System Design Series - Scalability

STEVE — Sat, 29 Jun 2024 22:07:52 +0000

System Design Series - Scalability

Introduction

In this section, we are going to discuss scalability, a critical aspect of system design that ensures your application can handle increased load gracefully. Understanding scalability is essential for building robust, high-performance systems that can grow with user demand and business needs.

What is Scalability?

Scalability is the ability of a system to handle increased workload by adding resources. It ensures that as demand grows, the system can continue to function efficiently. Scalability can be thought of in three dimensions:

Vertical Scalability (Scaling Up): Adding more power (CPU, RAM, etc.) to an existing machine.
Horizontal Scalability (Scaling Out): Adding more machines to a system.
Diagonal Scalability: Combining both vertical and horizontal scaling.

Types of Scaling

Vertical Scaling (Scaling Up)

Description: Increasing the capacity of a single machine by adding more resources (CPU, RAM, storage).
Pros:
- Easier to implement since it involves upgrading existing machines.
- No need to modify the application architecture.
Cons:
- Limited by the maximum capacity of a single machine.
- Single point of failure: if the machine goes down, the application becomes unavailable.
Use Cases: Initial stages of a project, applications with low to moderate growth.

Horizontal Scaling (Scaling Out)

Description: Adding more machines to handle increased load.
Pros:
- Virtually limitless scalability by adding more machines.
- Increases redundancy, reducing the risk of a single point of failure.
Cons:
- More complex to implement due to the need for distributed systems design.
- Requires load balancing and data distribution strategies.
Use Cases: High-growth applications, distributed systems, applications requiring high availability.

Diagonal Scaling

Description: A combination of vertical and horizontal scaling. Start with vertical scaling and switch to horizontal scaling when the vertical limit is reached.
Pros:
- Flexibility to adapt to different stages of growth.
- Optimizes resource utilization.
Cons:
- Requires careful planning and monitoring to switch between scaling strategies effectively.
Use Cases: Applications with varying load patterns, systems with mixed workloads.

Auto Scaling

Description: Automatically adjusting the number of running instances based on current load.
Pros:
- Dynamic scaling based on real-time demand.
- Cost-efficient as resources are used only when needed.
Cons:
- Requires accurate load prediction and monitoring.
- Potential for delays in scaling actions, leading to temporary performance issues.
Use Cases: Cloud-based applications, unpredictable traffic patterns, cost-sensitive applications.

Key Considerations for Scalability

Load Balancing

Load balancing distributes incoming network traffic across multiple servers, ensuring no single server becomes a bottleneck. Popular load balancers include:

Hardware Load Balancers: Physical devices designed to distribute traffic.
Software Load Balancers: Tools like Nginx, HAProxy, and cloud-based solutions like AWS Elastic Load Balancing.

Caching

Caching reduces the load on your servers by storing frequently accessed data in memory. Types of caching include:

Client-Side Caching: Caching data on the user's device.
Server-Side Caching: Caching data on the server side using tools like Redis or Memcached.
Content Delivery Networks (CDNs): Caching static assets (images, videos, etc.) on servers closer to the user.

Database Scaling

Databases can be a significant bottleneck in scalable systems. Techniques for scaling databases include:

Read Replicas: Distributing read requests to multiple read-only copies of the database.
Sharding: Partitioning the database into smaller, more manageable pieces called shards.
NoSQL Databases: Databases like MongoDB, Cassandra, and DynamoDB are designed for horizontal scaling.

Microservices Architecture

Microservices architecture breaks down a monolithic application into smaller, independent services that can be developed, deployed, and scaled independently. This approach enhances scalability by allowing individual services to be scaled based on their specific demands.

Auto Scaling

Auto scaling automatically adjusts the number of running instances based on current load. Cloud providers like AWS, Google Cloud, and Azure offer auto scaling features, ensuring your application scales dynamically in response to traffic changes.

Real-World Examples

Vertical Scaling Example: A startup begins with a single server for their web application. As they gain more users, they upgrade the server’s RAM and CPU to handle the increased load. This works well initially but eventually reaches a hardware limit.
Horizontal Scaling Example: A popular e-commerce site handles millions of users during peak seasons. They use multiple servers behind a load balancer to distribute incoming traffic. If one server fails, the load balancer redirects traffic to the remaining servers, ensuring uninterrupted service.
Diagonal Scaling Example: A SaaS company starts with vertical scaling by upgrading their servers as their user base grows. When they reach the limit of vertical scaling, they transition to horizontal scaling by adding more servers and implementing load balancing.
Auto Scaling Example: A news website experiences fluctuating traffic with sudden spikes during breaking news. Using auto scaling, the website dynamically adjusts the number of servers to handle the traffic spikes, ensuring consistent performance and cost efficiency.

Conclusion

Scalability is a fundamental aspect of system design that ensures your application can handle growth efficiently. By understanding and implementing various scaling techniques, from vertical, horizontal, and diagonal scaling to auto scaling, load balancing, caching, and database strategies, you can build systems that perform well under increased load.

Remember, the right scaling strategy depends on your specific use case, and often, a combination of methods will yield the best results. In the next section of our System Design Series, we will delve deeper into load balancing techniques and their importance in building scalable systems.

Node.js and GraphQL Tutorial: How to build a GraphQL API with an Apollo server

STEVE — Sat, 26 Aug 2023 16:04:42 +0000

INTRODUCTION
Welcome to my tutorial on building a GraphQL API with an Apollo Server using Node.js! In this guide, I will show you how to harness the power of GraphQL to create efficient and flexible APIs for your applications.

So, what exactly is GraphQL? Imagine a world where you can request exactly the data you need from your server, no more and no less. GraphQL is a query language that allows you to do just that. Unlike traditional REST APIs where you often receive a fixed set of data in predefined endpoints, GraphQL empowers you to shape your queries to match your specific requirements. It's like having a tailor-made API at your fingertips.

Comparing GraphQL to REST is like comparing a custom-made suit to off-the-rack clothing. With REST, you might end up over-fetching or under-fetching data, causing inefficiencies. But with GraphQL, you have the freedom to ask for only the fields you need, eliminating unnecessary data transfer and optimizing your app's performance.

But that's not all! GraphQL also excels in its ability to consolidate multiple data sources into a single query. No more juggling between different endpoints to assemble the data you need. GraphQL brings it all together in one elegant request.

Whether you're building a simple to-do app or a complex e-commerce platform, GraphQL's flexibility and efficiency can revolutionize the way you interact with APIs. Throughout this tutorial, I'll guide you step by step in creating a GraphQL API using an Apollo Server with Node.js. You'll learn how to define your data schema and fetch data from MongoDB using resolvers for a seamless experience.

So, if you're ready to dive into the world of GraphQL and unlock its potential, let's get started!

FOLDER STRUCTURE
When diving into building a GraphQL API with an Apollo Server, having a clear and organized folder structure is key. Here's a suggested layout for your project's folder structure, drawing parallels to how things might be organized in a traditional RESTful API project.

project-root/
|-- src/
|   |-- schema/           # GraphQL schema definitions
|   |-- resolvers/        # Resolver functions for handling queries and mutations
|   |-- models/           # Data models or database schemas
|   |-- app.js            # GraphQL server setup
|-- package.json          # Project dependencies and scripts
|-- .gitignore            # Git ignore configurations
|-- .env                  # Environment variables (optional)

Here's what each folder represents:

schema: Think of the "schema" folder as similar to the routes or endpoints in a RESTful API. Here, you define your GraphQL schema using a Schema Definition Language (SDL). This schema outlines the types, queries, mutations, and even subscriptions that your API will support. It's the heart of your API's structure.
resolvers: Just as controllers handle the logic for different routes in a RESTful API, resolvers in the "resolvers" folder handle the logic for various fields in your GraphQL schema. Each resolver function corresponds to a specific field and contains the actual code that fetches data, interacts with databases, and performs the required operations. Resolvers are where the "magic" happens, similar to how controllers execute the actions in a RESTful API.
models: The "models" folder houses your data models, which are equivalent to the database schema in a RESTful API context. These models define the structure and relationships of your data. In resolvers, you utilize these models to interact with your data sources, just as you would in a RESTful API's database layer.
app.js: The "app.js" file takes on the role of setting up and configuring your GraphQL server. This is equivalent to the server configuration and middleware setup you might do in the main file of a RESTful API.
package.json: This file remains the same regardless of whether you're working with REST or GraphQL. It lists your project's dependencies and scripts for managing your API, much like in a RESTful API project.
.gitignore: Similar to a RESTful API project, the ".gitignore" file helps you specify files and directories that should be ignored by version control (Git).
.env: Optionally, you can use the ".env" file to store environment variables for your application, just like in a RESTful API.

Remember, while this suggested folder structure draws parallels to RESTful design, it's adaptable to your project's specific needs and your development preferences. It provides a roadmap for organizing your GraphQL API project effectively, leveraging concepts you might already be familiar with from working with RESTful APIs.

PACKAGES
For this project, you will need to install the following packages:
mongoose, bcryptjs, dotenv, @apollo/server @graphql-tools/merge

Let's briefly talk about these packages:

mongoose: ODM library for MongoDB and Node.js, aiding data modeling and interaction.
bcryptjs: Library for secure password hashing and comparison.
dotenv: Loads environment variables from .env files, ensuring secure configuration.
@apollo/server: GraphQL server implementation for streamlined schema execution and validation.
@graphql-tools/merge: Utility for combining multiple GraphQL schemas into one cohesive schema.

Because I am using typescript, my folder structure and package.json file now look like this.

Now update the ./src/db/connect.ts file with the code below to establish a connection to your Mongo database.

import mongoose from "mongoose";

export const connectDB = (url : string) => {
    return mongoose.connect(url)
    .then(() => console.log("Connected to database"))
    .catch((err) => console.log(err));
}

For this project, I created two models- User and Products. This is what they look like:

./src/model/user.ts

import {Schema, Model, model, Document} from 'mongoose';
import bcrypt from 'bcryptjs';

export interface IUser extends Document {
    username: string;
    email: string;
    password: string;
    isValidPassword: (password: string) => Promise<boolean>;
}

const UserSchema: Schema = new Schema({
    username: {type: String, required: true, unique: true},
    email: {type: String, required: true},
    password: { type: String, required: true}
})

UserSchema.pre('save', async function() {
    const salt = await bcrypt.genSalt(10);
    this.password = await bcrypt.hash(this.password, salt);
})

UserSchema.methods.isValidPassword = async function(password: string) {
    const compare = await bcrypt.compare(password, this.password);
    return compare;
}

export const User: Model<IUser> = model<IUser>('User', UserSchema);

./src/model/product.ts

import {Schema, Model, model, Document} from 'mongoose'

export interface IProduct extends Document {
    name: string;
    price: number;
}

const ProductSchema: Schema = new Schema({
    name: {type: String, required: true},
    price: {type: Number, required: true}
})

export const Product: Model<IProduct> = model<IProduct>('Product', ProductSchema);

Now that we have successfully created a database schema for our API let us head straight to create the schema for our GraphQL queries and Mutations.

For the users schema, update ./src/schema/user.ts to look like this:

import { buildSchema } from "graphql";

export const usersGQLSchema = buildSchema(`
    type User {
        id: String!
        username: String!
        email: String!
        password: String!
    }

    type Query {
        users: usersInfoResponse!
        user(id: String!): User!
    }

    type usersInfoResponse {
        success: Boolean!
        total: Int!
        users: [User!]!
    }

    type Mutation {
        regUser(username: String!, email: String!, password: String!): User!
        loginUser(email: String!, password: String!): User!
        updateUser(id: String!, username: String, email: String, password: String): User!
        deleteUser(id: String!): deleteResponse!
    }

    type deleteResponse {
        success: Boolean!
        message: String!
        id: String!
    }

`)

Let me break down everything this code presents:

1. Types - Like Data Structures
Type User: Just as in RESTful APIs, a "type" in GraphQL defines what data looks like. For instance, "User" is like a blueprint for a user's data, including properties such as ID, username, email, and password.

2. Queries - Retrieving Data
Query users: Think of this as a way to request a list of users. Similar to a RESTful API endpoint, you're asking for user information. The exclamation point (!) after usersInfoResponse indicates that this query always returns a response with users' information.

Query user(id): This is like getting details about one user, just as you would in a RESTful API by providing an ID. The id parameter is marked with an exclamation point (!) to show that it's required for the query to work. The exclamation point after User indicates that this query always returns user information.

3. Mutations - Modifying Data
Mutation regUser/loginUser: Similar to creating a new resource in REST, this mutation lets you signup/signin a new user. The exclamation points after username, email, and password indicate that these fields are required for creating a user. The exclamation point after User indicates that the mutation always returns the newly created user's information.

Mutation updateUser(id): This is like updating a user's information, comparable to editing a resource in REST. The id is required, and you can modify username, email, or password. If you don't provide an exclamation point, it means the field is optional.

Mutation deleteUser(id): Just as you might delete a resource in REST, this mutation removes a user. The id is required, and the exclamation point after deleteResponse indicates that it always returns a response.

4. Custom Response Types - Structured Responses
Type usersInfoResponse: This is like the response you might get when requesting a list of users in REST. The exclamation point after success, total, and users means that these fields are always included in the response.

Type deleteResponse: Comparable to a response when deleting a resource in REST, this type always includes success, message, and id.

In essence, GraphQL's exclamation points highlight required fields just like in RESTful APIs. They ensure that when you make a query or mutation, you get back the data you need with certainty.

After creating our User schema, let us create the resolver (or controllers) to implement the logic.

Update ./src/resolver/user.ts with this code:

import {User} from '../model/user';

interface Args {
    id: string;
    username: string;
    email: string;
    password: string;
}

export const UsersResolver = {
    Query : {
        users: async () => {
            try {
                const users = await User.find({});
                if (!users) throw new Error('No users found');
                return {
                    success: true,
                    total: users.length,
                    users
                };
            } catch (error) {
                throw error;
            }
        },    

        user: async (_ : any, args : Args) => {
            try {
                if (!args.id) throw new Error('No id provided');
                const user = await User.findById(args.id);
                if (!user) throw new Error('No user found');
                return user;
            } catch (error) {
                throw error;
            }
        }
    },

    Mutation : {
        regUser: async (_ : any, args : Args) => {
            try {
                const user = await User.findOne({email: args.email});
                if (user) throw new Error('User already exists');
                const newUser = await User.create({
                    username: args.username,
                    email: args.email,
                    password: args.password
                })
                return newUser;
            } catch (error) {
                throw error;
            }
        },

        loginUser: async (_ : any, args : Args) => {
            try {
                const user = await User.findOne({email: args.email});
                if (!user) throw new Error('User not found');
                const isValid = await user.isValidPassword(args.password);
                if (!isValid) throw new Error('Invalid password');
                return user;
            } catch (error) {
                throw error;
            }
        },

        updateUser: async (_ : any, args : Args) => {
            try {
                const id = args.id;
                if (!id) throw new Error('No id provided');
                const user = await User.findById(args.id);
                if (!user) throw new Error('User not found');
                const updateUser = await User.findByIdAndUpdate(id, {...args}, {new: true, runValidators: true});
                return updateUser;
            } catch (error) {
                throw error;
            }
        },

        deleteUser: async (_ : any, args : Args) => {
            try {
                const id = args.id;
                if (!id) throw new Error('No id provided');
                const user = await User.findById(args.id);
                if (!user) throw new Error('User not found');
                const deleteUser = await User.findByIdAndDelete(id);
                return {
                    success: true,
                    message: 'User deleted successfully',
                    id: deleteUser?._id
                };
            } catch (error) {
                throw error;
            }
        }
    }
}

In the code above:

The UsersResolver object is created, containing resolvers for both queries and mutations.

The Query section contains two resolvers:

users: Fetches all users from the database and returns a response containing information about users.
user: Fetches a user by their provided ID from the database.
The Mutation section contains several resolvers for various operations, each performing a specific action:

regUser: Registers a new user if they don't already exist in the database.
loginUser: Validates user credentials and returns the user if login is successful.
updateUser: Updates a user's information based on their ID.
deleteUser: Deletes a user by their ID.
Each resolver uses asynchronous code (async/await) to interact with the database and handle potential errors.

The Args interface defines the expected arguments for the resolver functions. For instance, each mutation requires id, username, email, and password parameters.

This code demonstrates how GraphQL resolvers work to fetch, create, update, and delete data while interacting with a User model from an external module. It's similar to the logic you might use in controllers for RESTful APIs, where each resolver corresponds to a specific API operation.

Great work, now that we have successfully implemented the logic for Users let us repeat the same process by creating a schema and a resolver for Products.

./src/schema/products.ts

import {buildSchema} from "graphql"

export const productsGQLSchema = buildSchema(`
    type Product {
        id: String!
        name: String!
        price: Int!
    }

    type Query {
        products: productsInfoResponse!
        product(id: String!): Product!
    }

    type productsInfoResponse {
        success: Boolean!
        total: Int!
        products: [Product!]!
    }

    type Mutation {
        addProduct(name: String!, price: Int!): Product!
        updateProduct(id: String!, name: String, price: Int): Product!
        deleteProduct(id: String!): deleteResponse!
    }

    type deleteResponse {
        success: Boolean!
        message: String!
        id: String!
    }
`)

./src/resolvers/products.ts

import { Product } from "../model/products";

interface Args {
    id: string;
    name: string;
    price: number;
}

export const ProductsResolver = {
    Query : {
        products: async () => {
            try {
                const products = await Product.find({});
                if (!products) throw new Error('No products found');
                return {
                    success: true,
                    total: products.length,
                    products
                };
            } catch (error) {
                throw error;
            }
        },

        product: async (_ : any, args : Args) => {
            try {
                if (!args.id) throw new Error('No id provided');
                const product = await Product.findById(args.id);
                if (!product) throw new Error('No product found');
                return product;
            } catch (error) {
                throw error;
            }
        }
    },

    Mutation : {
        addProduct: async (_ : any, args : Args) => {
            try {
                const product = await Product.findOne({name: args.name});
                if (product) throw new Error('Product already exists');
                const newProduct = await Product.create({
                    name: args.name,
                    price: args.price
                })
                return newProduct;
            } catch (error) {
                throw error;
            }
        },

        updateProduct: async (_ : any, args : Args) => {
            try {
                const id = args.id;
                if (!id) throw new Error('No id provided');
                const product = await Product.findById(args.id);
                if (!product) throw new Error('No product found');
                const updateProduct = await Product.findByIdAndUpdate(id, {...args}, {new: true, runValidators : true});
                return updateProduct;
            } catch (error) {
                console.log(error)
            }
        },

        deleteProduct: async (_ : any, args : Args) => {
            try {
                const id = args.id;
                if (!id) throw new Error('No id provided');
                const product = await Product.findById(args.id);
                if (!product) throw new Error('No product found');
                const deleteProduct = await Product.findByIdAndDelete(id);
                return {
                    success: true,
                    message: 'Product deleted successfully',
                    id: deleteProduct?._id
                };
            } catch (error) {
                throw error;
            }
        }
    }
}

Similar to the implementation for User entities, the schema and resolver for Product entities collaborate seamlessly to offer a comprehensive and structured approach to managing product-related operations. This cohesive interaction ensures that querying, creating, updating, and deleting products within your GraphQL API are handled efficiently and logically.

The product schema serves as a blueprint, defining the structure of a Product type. Just as with the User type, this schema outlines the fields that compose a product, such as ID, name, description, price, and more. It specifies not only the fields themselves but also their data types and whether they're required or optional.

On the other hand, the product resolver takes care of the functional aspects. In a manner akin to the user resolver, it encapsulates the actual logic behind queries and mutations involving products. For instance, when querying for a list of products, the resolver fetches the products from a data source (e.g., a database. In our case - MongoDB) and constructs a well-formed response with details about each product. Similarly, when creating, updating, or deleting a product, the resolver handles the necessary data manipulation, validation, and interaction with the data source.

In tandem, the schema and resolver form a cohesive unit that makes it straightforward to understand, implement, and maintain product-related operations in your GraphQL API. This separation of concerns between defining the structure (schema) and implementing the functionality (resolver) contributes to a clean and organized codebase, making your API development experience smoother and more structured.

Now it's time for us to combine all the Schema and resolvers so that we can import a merged type definition and schema into the app.ts file.

Update ./src/schema/index.ts

import {mergeTypeDefs} from "@graphql-tools/merge"

import { usersGQLSchema } from "./user"
import { productsGQLSchema } from "./products"

export const mergedGQLSchema = mergeTypeDefs([usersGQLSchema, productsGQLSchema])

By doing this, you're creating a unified GraphQL schema that incorporates all the types and operations defined in the user and product schemas. This merged schema can then be imported into your app.ts file to create a cohesive GraphQL API that supports both user and product-related functionalities.

Update ./src/resolver/index.ts

import { UsersResolver } from "./user";
import { ProductsResolver } from "./product";

export const resolvers = [UsersResolver, ProductsResolver]

When we set up your GraphQL server, you'll use this combined array of resolvers along with the merged schema to create a fully functional GraphQL API.

We're nearing completion, and the final step is to construct our app.ts file, where we'll consolidate the various components we've developed so far. This file will serve as the backbone of our GraphQL application.

To begin, we load environment variables from a .env file using the dotenv library, ensuring the secure configuration of sensitive data like database connection details and port numbers.

Importing essential dependencies follows suit. These include the function responsible for connecting to the database (connectDB), as well as the ApolloServer and startStandaloneServer modules that facilitate GraphQL server creation.

Within this context, we define a constant named PORT to encapsulate the port number for our server. This value is either extracted from environment variables or defaults to 3000.

Our ApolloServer instance takes center stage. We configure it with the merged GraphQL schema (mergedGQLSchema) and the combined resolvers (resolvers). Moreover, we enable introspection, a valuable tool for introspecting the schema using tools like GraphQL Playground.

To bring it all to life, the start function emerges. As an asynchronous function, it orchestrates the setup process:

It employs the connectDB function to establish a connection to the MongoDB database, using the URI from environment variables.
The startStandaloneServer function is then invoked, initiating the Apollo server's operation. This server listens attentively on the specified port (PORT).
The culmination of this sequence is marked by a console message announcing the server's successful launch.

With the completion of these steps, we culminate the process by invoking the start function. This action ignites the journey, connecting the database, and propelling the GraphQL server into operation.

This is what our app.ts file looks like:

require("dotenv").config()

import { connectDB } from "./db/connect";

import { ApolloServer } from '@apollo/server';

import { startStandaloneServer } from '@apollo/server/standalone';

import { mergedGQLSchema } from "./schema";
import { resolvers } from "./resolvers";

const PORT = parseInt(process.env.PORT as string) || 3000

const server = new ApolloServer({
    typeDefs : mergedGQLSchema,
    resolvers : resolvers,
    introspection : true
  });

const start = async () => {
    try {
        connectDB(process.env.MONGO_URI as string)
        startStandaloneServer(server, { listen: { port: PORT } });
        console.log(`Server is listening on port ${PORT}`)
    } catch (error) {
        console.log(error)
    }
}

start()

And with that, we're all set. To set the server in motion, simply execute the command npm run dev or npm start. Afterwards, pinpoint the endpoint at http://localhost:3000. If the components fall into place as anticipated, your browser will transport you to the interactive Apollo GraphQL Playground.

This serves as an excellent feature as GraphQL is inherently self-documenting. The GraphQL Playground, an integrated query tool, allows you to test and construct queries with ease. Much like the functionality found in tools like Postman, you can formulate queries, explore the schema, and gain insights into your API's capabilities firsthand.

It would look like this:

Query to get all users:

Mutation to register a user:

Mutation to update a product

Notice that all endpoints are accessible via http://localhost:3000/, unlike a RESTful design where you would need to define different endpoints for each route. This is because of GraphQL's single endpoint architecture and its ability to handle complex data retrieval in a more dynamic manner.

In a traditional RESTful API design, each endpoint typically corresponds to a specific resource or route. If you wanted to retrieve different types of data, you would need to create distinct endpoints for each resource. For example, to fetch user information, you might have an endpoint like GET /users, and for products, another endpoint like GET /products.

However, GraphQL takes a different approach. With GraphQL, there's a single endpoint that serves as the entry point for all data operations. This endpoint is usually accessed via an HTTP POST request. Instead of defining multiple endpoints for different resources, GraphQL employs a flexible querying system that allows you to request exactly the data you need, and nothing more.

This is where the power of the GraphQL query language shines. The client can specify the shape and structure of the data it requires by creating queries that match the types and fields defined in the GraphQL schema. It's like asking for a custom-made data response tailored to your application's needs.

Behind the scenes, the GraphQL server processes the query and retrieves only the requested data. This eliminates the need to create and manage numerous endpoints for different use cases. The single endpoint approach simplifies the API structure, reduces redundancy, and provides a more efficient way to interact with data.

In essence, GraphQL's single endpoint design, coupled with its dynamic querying capabilities, offers a more streamlined and adaptable approach to handling complex data retrieval compared to the more rigid endpoint structure of traditional RESTful APIs. This contributes to the efficiency and flexibility that GraphQL brings to modern API development.

CONCLUSION
In conclusion, GraphQL presents a host of advantages that make it a compelling choice for modern API development. Its flexible querying system empowers clients to precisely request only the data they need, eliminating over-fetching and under-fetching of data commonly associated with RESTful APIs. This optimization in data transfer enhances performance, reduces unnecessary network traffic, and results in faster, more efficient interactions.

Unlike RESTful APIs that often require multiple endpoints for distinct resources, GraphQL's single endpoint architecture simplifies API management and reduces the need for versioning. With GraphQL, you have the freedom to evolve your API without causing disruption to existing clients, as fields can be added or deprecated without changing the endpoint structure.

Furthermore, GraphQL's introspection capabilities grant developers access to in-depth schema documentation, making it a self-documenting API. This, coupled with the integrated query tools like the GraphQL Playground, streamlines development, debugging, and testing.

However, it's essential to acknowledge that GraphQL might not be the optimal solution for every scenario. Its flexibility might lead to complex queries, potentially putting a heavier load on servers. RESTful APIs, on the other hand, can offer a clearer mapping to underlying data models and cache management due to their predictable nature.

In comparison, GraphQL and RESTful APIs each have their strengths and weaknesses, catering to different project requirements. While GraphQL excels in scenarios that prioritize flexibility, efficient data retrieval, and a unified endpoint, RESTful APIs can be more suitable for situations where a clear, resource-oriented structure and caching mechanisms are vital.

In the journey of building GraphQL APIs, we've traversed the process of creating schemas, defining types, crafting resolvers, and setting up the server. The completed code for this tutorial is accessible on Github via GRAPHQL-API, offering a practical reference for your endeavors.

To all the readers embarking on this exploration, I extend my appreciation for joining this tutorial. Whether you choose GraphQL or RESTful APIs, may your coding journeys be filled with innovation, efficiency, and transformative experiences. Happy coding!

HOW TO AUTOMATE CI/CD ON YOUR AZURE KUBERNETES CLUSTER

STEVE — Tue, 15 Aug 2023 20:25:39 +0000

INTRODUCTION

In the world of modern software development, delivering applications rapidly and reliably is paramount. Continuous Integration and Continuous Deployment (CI/CD) practices streamline the development lifecycle, enabling teams to automate building, testing, and deploying applications. When coupled with the power of Kubernetes, a robust container orchestration platform, the efficiency and scalability of your applications reach new heights.

In this comprehensive tutorial, I will walk you through the process of automating CI/CD for your applications on an Azure Kubernetes Service (AKS) cluster. You'll learn how to set up a complete pipeline that connects your GitHub repository to your AKS cluster, enabling automatic building, testing, and deployment of your containerized applications. Whether you're new to Kubernetes and CI/CD or looking to refine your skills, this guide has you covered.

Prerequisites:
Before diving into the tutorial, ensure you have the following prerequisites in place:

GitHub Account: You'll need an active GitHub account to host your application's source code and set up the pipeline for CI/CD.
Azure Account: You'll require an Azure account with either a free subscription or a pay-as-you-go subscription. If you're new to Azure, you can take advantage of the $200 free trial credit for the first month to explore and experiment with AKS and other Azure services.
Azure DevOps Account: To seamlessly integrate your CI/CD pipeline, an Azure DevOps account is necessary. This account will allow you to configure the automation process and manage the flow of changes from source code to the AKS cluster.

Outline:
Throughout this tutorial, we'll cover the following key topics:

Creating an Azure Kubernetes Cluster:
- Understand the benefits of using AKS for container orchestration.
- Step-by-step guide to creating an AKS cluster in your Azure account.
- Exploring AKS features and configurations.
Setting Up a GitHub Pipeline for Docker and Kubernetes:
- Introduction to CI/CD and its importance in modern development.
- Configuring your GitHub repository for seamless integration with Azure DevOps.
- Creating a CI/CD pipeline that automates Docker image builds and Kubernetes deployments to your AKS cluster.

By the end of this tutorial, you'll have gained practical insights into the world of CI/CD automation on Azure Kubernetes Service, empowering you to accelerate your software delivery process while maintaining high standards of reliability and efficiency.

So, let's embark on this journey to unlock the potential of automating CI/CD on your Azure Kubernetes Cluster. Ready to get started? Let's dive in!

First things first we have to build our docker image locally. I have set up a simple Typescript-Nodejs server with a few routes: home, about, contact, and a universal 404. (This can be setup using any framework) Here is a link to my code on GitHub - CODE.
Here is what the routes look like :

I have also set up a basic Docker file configuration. Here is what it looks like :

Now I will simply build a new image using this command :

docker build -t kubernetes-pipeline .

Then ensure the docker image is running using this command :

docker run -d --name kubernetes-pipeline -p 3000:3000 kubernetes-pipeline

This should start the docker container on port 3000

CREATING A REGISTRY
Next, we need to push this image to a registry on Microsoft Azure. This will be a good time to create an Azure account if you don't already have one. You can create an Azure account here: Azure.

Once your account has been created successfully, in the search bar that appears on the Azure dashboard, search for registries and select Create container registry.

Once you have successfully created your registry, we will proceed to push our image to the registry using this command:

Tag image to the repository: docker tag kubernetes-pipeline onlyregistryhere.azurecr.io/kubernetes-pipeline:latest

Push the image to Azure registry: docker push onlyregistryhere.azurecr.io/kubernetes-pipeline:latest

Ensure to replace onlyregistryhere with your registry name and kubernetes-pipeline:latest with your docker image name and tag.

If everything works as expected, you should see your image name in the list of repositories.

CREATING AN AZURE KUBERNETES CLUSTER
Now let us create our Kubernetes cluster. From the Azure dashboard, simply search Kubernetes services, follow the prompt, and create a cluster. If everything has been set up correctly, you should see this:

To interact with our cluster and manage services and deployments, I recommend using the Cloud shell. Therefore, locate the cloud shell in the get started menu and click on connect.
Once you open the cloud shell, you can now interact with all the pods we will deploy in the future. save this tab and let us head over to Azure DevOps.

Our cloud shell should look like this:

CI WITH AZURE DEVOPS
Now, to automate the CI/CD process, you need to have an Azure DevOps account. If you don't, simply head over to Azure DevOps to create a free account.

Next, click on create a new project. Once created, locate pipeline and select Github/Gitlab to create a pipeline with your chosen host. This may prompt you to authorize this action from your Github/Gitlab account. After authorization, select the repository you would like to create a CI pipeline for and select okay. Next, configure your pipeline from the list of available options. In our case, we will select Docker to build and push the image to Azure Container Registry, then we will later run a separate pipeline - Deploy to Azure Kubernetes Service to create a pipeline for your Azure Kubernetes Service. follow the default prompts and grant all the necessary permissions to configure a successful CI pipeline.

If everything checks out, you should see this:

Now to confirm that our deployment works fine, let us head back to our cloud shell and query for all services and deployments using the following commands:

kubectl get deployments: get all deployments.
kubectl get services: get all services.

Let me explain what is happening in the cloud shell.

kubectl get deployments:
- Deployment name kubernetespipeline.
- 1 pod is ready and available out of 1.
- Deployment is up-to-date.
- Age: 56 seconds.
kubectl get services:
- kubernetes service (core Kubernetes service):
  - ClusterIP: 10.0.0.1.
  - No external IP.
  - Type: ClusterIP.
  - Age: 129 minutes.
- kubernetespipeline service (created service):
  - ClusterIP: 10.0.25.184.
  - External IP: 51.142.173.251.
  - Type: LoadBalancer.
  - Port Mapping: 3000:31553.
  - Age: 60 seconds.
kubectl get pods:
- Pod name: kubernetespipeline-5d677f89c8-qvcsp.
- 1/1 containers in the pod are ready.
- Pod status: Running.
- No restarts.
- Age: 2 minutes.
kubectl logs -f kubernetespipeline-5d677f89c8-qvcsp:
- Following logs for pod kubernetespipeline-5d677f89c8-qvcsp.
- Application: kubernetes-pipeline@1.0.0, started with node ./dist/app.js.
- Server running on port 3000.
- Logs show requests to "Home page!", "Contact page!", and "About page!".

The output provides a snapshot of the Kubernetes deployment, services, pod, and application logs in my cluster. A deployment named kubernetespipeline has a ready pod, and a service named kubernetespipeline is externally accessible. The pod, kubernetespipeline-5d677f89c8-qvcsp, is running an application serving requests on port 3000, with logs indicating various page accesses. The core kubernetes service and its details are also displayed.

Now let me access this endpoint on my local browser.

Conclusion:
See? You've successfully accessed your containerized API using the external IP of your service within your Kubernetes cluster. By following the steps outlined in this tutorial, you've not only set up a seamless integration between your GitHub repository and your Kubernetes cluster but also automated the process of updating your application. This streamlined approach gives you more time to concentrate on what truly matters – the development of your application itself.

What's Next?
The journey doesn't stop here. You've established a robust foundation for your CI/CD pipeline, but there are more enhancements and optimizations you can explore:

Custom Domain Setup:
Take your application to the next level by providing a custom domain for your deployment. This way, users can access your API using a memorable and branded URL. You can achieve this by setting up an Ingress controller in Kubernetes and configuring it to route traffic to your service. This enhances user experience and aligns with professional standards.
Scale and Load Balancing:
As your application gains popularity and user traffic increases, you can further optimize performance by exploring Kubernetes' scaling and load balancing capabilities. Configure Horizontal Pod Autoscaling to dynamically adjust the number of pods based on traffic load, ensuring smooth user experiences during traffic spikes.
Security and Authentication:
Protect your API and user data by implementing security measures. Explore Kubernetes' built-in security features, like Network Policies, to control communication between pods. Additionally, consider integrating authentication and authorization mechanisms to ensure that only authorized users can access your API.
Monitoring and Logging:
Gain insights into your application's behavior and performance by setting up monitoring and logging solutions. Tools like Prometheus and Grafana can help you monitor resource usage and visualize metrics, enabling you to proactively address any issues.

As you venture further into the realm of Kubernetes, CI/CD, and application development, remember that your learning journey is ongoing. Embrace new challenges and keep exploring advanced techniques to create more efficient, reliable, and user-friendly applications.

Farewell:
With that, I bid farewell to this tutorial. I hope that this guide has provided you with a solid foundation to automate your CI/CD pipeline on an Azure Kubernetes Service cluster. Remember, technology evolves, and so does your expertise. Keep experimenting, learning, and innovating, and you'll continue to build amazing solutions that make a real impact.

Thank you for joining me on this journey, and best of luck with your future endeavors in the exciting world of DevOps!

DOCKER FOR EVERYONE - (Learn about Caching, Load-Balancing, and Virtual Machines).

STEVE — Mon, 24 Jul 2023 13:45:01 +0000

INTRODUCTION

Hello there and welcome to this comprehensive tutorial on Docker, where I will be guiding you through the exciting world of load-balancing, caching, and deploying Docker containers to cloud services. Whether you're a beginner or an experienced developer, this tutorial is designed to be accessible and beneficial for everyone.

In this tutorial, we will cover a range of fundamental concepts and practical techniques to enhance your Docker skills. First and foremost, we'll delve into the basics of Docker and containerization, helping you understand the core principles and advantages of this powerful technology.

One of the key topics we'll explore is caching user sessions using Redis. Redis is an open-source, in-memory data structure store that allows for lightning-fast data retrieval, making it an ideal tool for caching frequently accessed data, like user sessions. I will guide you through the process of integrating Redis into your Docker workflow to optimize the performance of your applications.

Another critical aspect we'll address is load balancing using Nginx. Nginx is a high-performance web server that excels at distributing incoming network traffic across multiple endpoints. By incorporating Nginx into your Docker environment, you can effectively distribute the workload, ensuring smooth and efficient handling of incoming API requests.

Finally, we'll cover deploying your Docker containers to Microsoft Azure or your preferred cloud service. The ability to deploy applications to the cloud offers numerous benefits, including scalability, reliability, and easy access from anywhere. I'll provide step-by-step instructions to facilitate a seamless deployment process.

Before we begin, the only prerequisite for this tutorial is having a working API to follow along. If you don't have your own API, don't worry! You can simply clone my repository on Github by following this link : DOCKER, which we'll use throughout the tutorial.

I am committed to making this tutorial as comprehensive and informative as possible, hence the title "Docker for everyone." However, if you encounter any challenges along the way, feel free to reach out to me via the comment section. Additionally, don't hesitate to use online resources to overcome any roadblocks you may encounter during your learning journey.

What is Docker?

Docker is a powerful tool that provides a standardized and efficient way to package, distribute, and run applications. It addresses several challenges faced in traditional software development and deployment processes. including but not limited to the following.

Compatibility Issues: Docker ensures consistent behavior across different environments by encapsulating applications and their dependencies within containers. This eliminates compatibility issues that arise due to differences in operating systems, libraries, and configurations.
Dependency Management: With Docker, developers define application dependencies in a Dockerfile, and Docker takes care of including all required libraries and frameworks in the container image. This simplifies dependency management and ensures reproducible deployments.
Deployment Complexities: Docker's containerization simplifies application deployment, especially in complex setups with multiple microservices. It allows each service to run in its own container, making scaling, deployment, and management easier.
Scalability and Resource Utilization: Docker enables seamless application scaling through container orchestration platforms like Kubernetes or Docker Swarm. These platforms automatically adjust the number of containers based on demand, optimizing resource utilization and ensuring smooth user experiences.

In summary, Docker provides an efficient solution to challenges such as compatibility, dependency management, deployment complexities, and scalability, making it an essential tool for modern software development and deployment workflows.

Now that we've set the stage, let's dive into the fascinating world of Docker, load-balancing, caching, and virtual machines. Get ready to unlock the true potential of your applications with Docker's powerful capabilities.

As mentioned earlier, we will use a preexisting API. you can clone this repository from GitHub via this link : DOCKER

After cloning, you will need to supply the following environment variables :

From the environment variables above, we have included some credentials related to Redis. As mentioned earlier, we will use Redis to cache user sessions. so let me show you how we can include that in a typical Nodejs application. First, create a redis.js file in the config folder and populate it with the following code.

const { createClient } = require("redis");

const client = createClient({
    password: process.env.REDIS_PASSWORD,
    socket: {
        host: process.env.REDIS_HOST,
        port: process.env.REDIS_PORT,
    }
});

client.on("connect", () => {
    console.log("Connected to redis...")
})

client.on("error", (error) => {
    console.log("Error connecting to redis...", error)
})

module.exports = client

please ensure that you have a Redis instance running so that you can easily connect it to your Nodejs Application. visit Redis cloud to create a new Redis instance.

Now in the app.js, we will connect Redis to our API and use the express-session module to create sessions in our Redis database for our users.

This is the relevant code that achieves that purpose.

const redisClient = require("./config/redis");
const RedisStore = require('connect-redis').default;
const session = require('express-session');

// Initialize sesssion storage.
app.use(session({
  store: new RedisStore({ client: redisClient }),
  secret: process.env.SESSION_SECRET,
  name: 'express-session',
  cookie: {
    secure: false,
    httpOnly: true,
    maxAge: 60000, // 1 minute. you can extend the maxAge value to suite your needs.
    // You can also set other cookie options if needed.
  },
  resave: false, // Set this to false to prevent session being saved on every request.
  saveUninitialized: true, // Set this to true to save new sessions that are not modified.
}));

const start = async () => {
  try {
    await redisClient.connect() //connect API to redis
    await connectDB(mongoUrl || 'mongodb://localhost:27017/express-mongo');
    app.listen(PORT, () => {
      console.log(`Server is running on port ${PORT}`);
    });
  } catch (error) {
    console.log(error);
  }
};

start();

If everything works fine, our terminal should look like this :

And if we try to log in, we should see our cookie express-session in the cookie section.

After 1 minute the session should expire and you will get this error when you hit the get all users endpoint.

Great! Now that our cache works as expected, let us containerize our application. But before we dive into that, let's take a moment to familiarize ourselves with some essential keywords related to Docker:

Container: A container is a lightweight, isolated execution environment that contains an application and all its dependencies. It encapsulates the application, libraries, and configurations required to run the software. Containers provide consistency and portability, ensuring that the application runs consistently across different environments.
Image: An image is a read-only template used to create containers. It includes the application code, runtime, libraries, environment variables, and any other files required for the application to run. Docker images are the building blocks for containers.
Volume: A volume in Docker is a persistent data storage mechanism that allows data to be shared between the host machine and the container. Volumes enable data to persist even after the container is stopped or deleted, making it ideal for managing databases and other persistent data.
Dockerfile: A Dockerfile is a text file that contains instructions for building a Docker image. It specifies the base image, adds application code, sets environment variables, and defines other configurations needed for the container.
Dockerignore: The .dockerignore file is used to specify which files and directories should be excluded from the Docker image build process. This is useful to prevent unnecessary files from being included in the image and reduces the image size.
Docker Compose: Docker Compose is a tool for defining and managing multi-container Docker applications. It uses a YAML file to define the services, networks, and volumes required for the application to run. Compose simplifies the process of managing complex applications with multiple containers.
Services: In the context of Docker Compose, services refer to the individual components of a multi-container application. Each service represents a separate container running a specific part of the application, such as a web server, a database, or a cache.

Understanding these keywords will help you confidently move forward with containerizing your application using Docker. Let's explore how to utilize Docker to package our application into containers for seamless deployment and scalability.

First, as described above, create a Dockerfile in the root directory and populate it with the following code :

# specify the node base image with your desired version node:<version>
FROM node:16

WORKDIR /app

# copy the package.json to install dependencies
COPY package.json .

# install dependencies
RUN npm install

ARG NODE_ENV
RUN if [ "$NODE_ENV" = "development" ]; \
    then npm install; \
    else npm install --only=production; \
    fi

# copy the rest of the files
COPY . ./

# replace this with your application's default port
EXPOSE 3000

# start the app
CMD ["node", "app.js"]

Let's break down the configuration in the Dockerfile step by step:

FROM node:16: This line sets the base image for our Docker container. In this case, we are using the official Node.js Docker image with version 16 as our starting point. This base image includes the Node.js runtime and package manager, which we need to run our application.
WORKDIR /app: This line sets the working directory inside the container to /app. This is the directory where our application code will be copied and where we'll execute commands.
COPY package.json .: This line copies the package.json file from our local directory (the same directory as the Dockerfile) into the container's working directory. We do this first to take advantage of Docker's layer caching mechanism. It allows Docker to cache the dependencies installation step if the package.json file hasn't changed.
RUN npm install: This command runs the npm install command inside the container to install the application's dependencies listed in the package.json file. This ensures that all required packages are available inside the container.
ARG NODE_ENV: This line declares an argument named NODE_ENV. Arguments can be passed to the Docker build command using --build-arg option. It allows us to specify whether we are building the container for development or production environment.
RUN if [ "$NODE_ENV" = "development" ]; ...: This conditional statement checks the value of the NODE_ENV argument. If it is set to "development," it will run npm install again, installing the development dependencies. Otherwise, if NODE_ENV is set to anything other than "development" (e.g., "production"), it will only install production dependencies using npm install --only=production.
COPY . ./: This line copies all the files and directories from our local directory (the same directory as the Dockerfile) into the container's working directory (/app). This includes our application code, configuration files, and any other necessary files.
EXPOSE 3000: This instruction specifies that the container will listen on port 3000. It doesn't actually publish the port to the host machine; it's merely a way to document the port that the container exposes.
CMD ["node", "app.js"]: This sets the default command to be executed when the container starts. In this case, it runs the Node.js application using the node command with the entry point file app.js.

In summary, the Dockerfile is a set of instructions to build a Docker image for our Node.js application. It starts from the official Node.js image, sets up the working directory, installs dependencies based on the environment (development or production), copies our application code, specifies the exposed port, and defines the command to start our application. With this configuration, we can create a containerized version of our Node.js application that can be easily deployed and run consistently across different environments.

Next, let us create and populate three docker-compose files in the root directory of our application:

First docker-compose.yml file :

version: "3" # specify docker-compose version
services:
  nginx:
    image: nginx:stable-alpine # specify image to build container from
    ports:
      - "5000:80" # specify port mapping
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf # mount nginx config
  node-app:
    build: . # use the Dockerfile in the current directory
    environment:
      - PORT=3000 # container

Second docker-compose.dev.yml file :

version: "3"
services:
  nginx:
    image: nginx:stable-alpine # specify image to build container from
    ports:
      - "3000:80" # specify port mapping
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro # mount nginx config file
  node-app:
    build:
      context: . # current directory
      args:
        - NODE_ENV=development
    volumes:
      - ./:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
    command: npm run dev

Third docker-compose.prod.yml file :

version: "3"
services:
  nginx:
    image: nginx:stable-alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
  node-app:
    deploy:
      restart_policy:
        condition: on-failure
    build: 
      context: .
      args:
        - NODE_ENV=${NODE_ENV}
    volumes:
      - ./:/app
      - /app/node_modules
    command: npm start
    environment:
      - MONGO_USERNAME=${MONGO_USERNAME}
      - MONGO_PASSWORD=${MONGO_PASSWORD}
      - REDIS_HOST=${REDIS_HOST}
      - REDIS_PORT=${REDIS_PORT}
      - SESSION_SECRET=${SESSION_SECRET}
      - REDIS_PASSWORD=${REDIS_PASSWORD}
      - NODE_ENV=${NODE_ENV}

By using these docker-compose files, we can easily manage our containers and define different configurations for development and production environments. The combination of Docker and docker-compose simplifies the process of containerizing and deploying our application, making it more efficient and scalable in real-world scenarios.

Now let us break down the contents of all three files.

docker-compose.yml file:

The docker-compose.yml file is the main configuration file for our application. It allows us to define and manage multiple services, each running in its own container. Let's go through its contents:
version: "3": This line specifies the version of the docker-compose syntax that we are using. In this case, we are using version 3.
services: This section defines the different services (containers) that compose our application.
nginx: This service is responsible for running the Nginx web server.
image: nginx:stable-alpine: It specifies the base image for the nginx container, which will be pulled from Docker Hub. We are using the stable Alpine version of Nginx, a lightweight and efficient web server.
ports: This line maps port 5000 on the host machine to port 80 inside the nginx container. This allows us to access the Nginx server through port 5000 on our local machine.
volumes: Here, we mount the ./nginx/default.conf file from the host machine to the container's /etc/nginx/conf.d/default.conf path. This file is used to configure Nginx.
node-app: This service represents our Node.js application.
build: .: It tells Docker to build the node-app container using the Dockerfile located in the current directory (.).
environment: In this line, we set the PORT environment variable to 3000 inside the container. This variable allows our Node.js application to listen on port 3000.
These settings in the docker-compose.yml file allow us to run both Nginx and our Node.js application together, making them work seamlessly in tandem.

Now since we created a volume that mounts a custom nginx configuration in the docker-compose file, later on, we will need to create that file in our development environment and ensure we provide accurate configuration settings - (More on this later).

Next, we'll look at the other two docker-compose files used for different scenarios - development and production environments.

docker-compose.dev.yml file :

The docker-compose.dev.yml file is used for the development environment. It allows us to set up our application with configurations optimized for development purposes. Let's go through its contents:

version: "3": Same as in the previous file, this specifies the version of the docker-compose syntax used.
services: This section defines the services (containers) specific to the development environment.
nginx: This service runs the Nginx web server, just like in the previous file.
image: nginx:stable-alpine: The same base image for Nginx.
ports: Here, we map port 3000 on the host machine to port 80 inside the nginx container. This allows us to access the Nginx server through port 3000 on our local machine.
volumes: We mount the same ./nginx/default.conf file, but this time with the ro (read-only) option, as we don't need to modify it during development.
node-app: This service represents our Node.js application specifically for development.
build: It tells Docker to build the my-node-app container using the Dockerfile in the current directory (.). Additionally, we pass the NODE_ENV=development argument to the build process, allowing our application to use development-specific configurations.
volumes: Here, we mount the current directory (./) to the /app directory inside the container. This allows us to have real-time code changes reflected in the container without rebuilding it. We also mount /app/node_modules to prevent overriding the node_modules directory in the container and ensure our installed dependencies are available.
environment: We set the NODE_ENV environment variable to development inside the container to activate development-specific behavior in our Node.js application.
command: This line specifies the command to run when the container starts. In this case, we execute the npm run dev command, which usually starts our application in development mode.
The docker-compose.dev.yml file enables us to set up our development environment with the necessary configurations, ensuring the smooth and efficient development of our application.
Now, let's proceed to the last docker-compose file.

docker-compose.prod.yml file:

The docker-compose.prod.yml file is designed for the production environment. It defines the configurations optimized for running the application in a production setting, where reliability and scalability are crucial. Let's examine its contents:

version: "3": As before, this specifies the version of the docker-compose syntax used.
services: This section defines the services (containers) specific to the production environment.
nginx: This service runs the Nginx web server, just like in the previous files.
image: nginx:stable-alpine: The same base image for Nginx.
ports: Here, we map port 80 on the host machine to port 80 inside the nginx container, allowing HTTP traffic to reach the Nginx server on port 80.
volumes: Again, we mount the ./nginx/default.conf file, but this time with the ro (read-only) option, as we don't need to modify it during production.
node-app: This service represents our Node.js application specifically for production.
deploy: This section specifies deployment-related configurations for the service.
restart_policy: We set the restart policy to "on-failure," which means the container will automatically restart if it fails.
build: Similar to previous files, it tells Docker to build the node-app container using the Dockerfile in the current directory (.). Additionally, we use the NODE_ENV=${NODE_ENV} argument, allowing our application to use production-specific configurations.
volumes: We mount the current directory (./) to the /app directory inside the container, along with mounting /app/node_modules to preserve installed dependencies.
command: This line specifies the command to run when the container starts. In this case, we execute the npm start command, which usually starts our application in production mode.
environment: We set various environment variables (MONGO_USERNAME, MONGO_PASSWORD, REDIS_HOST, REDIS_PORT, SESSION_SECRET, REDIS_PASSWORD, and NODE_ENV) required by our Node.js application for production-specific settings.
The docker-compose.prod.yml file ensures that our application is optimally configured for a production environment, with reliability, scalability, and automatic restarts on failure. It allows us to deploy our application confidently, knowing that it is running efficiently and can handle real-world production scenarios.

At this point, we are almost done with the file setup, we now need to write our custom Nginx configuration to enable effective load-balancing within our container.

So create the Nginx config file to match the volume we declared in our docker-compose file - ./nginx/default.conf: now add the following lines of code :

upstream backend { # this is the name of the upstream block
    server using_docker_node-app_1:3000;
    server using_docker_node-app_2:3000;
    server using_docker_node-app_3:3000;
}

server {
    listen 80; # this is the port that the server will listen on

    location /api/ {
        proxy_set_header X-Real-IP $remote_addr; # this is required to pass on the client's IP to the node app
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # this is required to pass on the client's IP to the node app
        proxy_set_header Host $http_host; # this is required to pass on the client's IP to the node app
        proxy_set_header X-NginX-Proxy true; # this is required to pass on the client's IP to the node app
        proxy_pass http://backend; # this is the name of the upstream block
        proxy_redirect off; # this is required to pass on the client's IP to the node app
    }
}

Now let us explain this configuration :

upstream backend: This block defines an upstream group named "backend." It is used to define a list of backend servers that Nginx will load balance requests to. In this case, we have three servers (using_docker_node-app_1, using_docker_node-app_2, and using_docker_node-app_3) running our Node.js application on port 3000.

server: This block defines the server configuration for Nginx.

listen 80: This line specifies that the Nginx server will listen on port 80 for incoming HTTP requests.

location /api/: This block defines a location for Nginx to handle requests that start with /api/. We use this location to route requests to our backend Node.js application for API calls.

proxy_set_header: These lines set various headers to pass on information to the Node.js application:

X-Real-IP: Sets the client's IP address as seen by the Nginx server.
X-Forwarded-For: Appends the client's IP address to the X-Forwarded-For header, indicating the chain of proxy servers.
Host: Sets the original host header to preserve the client's hostname.
X-NginX-Proxy: Sets a header to indicate that the request is being proxied by Nginx.
proxy_pass http://backend;: This line directs Nginx to pass the incoming requests to the backend group named "backend" that we defined earlier. Nginx will automatically load balance the requests among the three servers specified in the "backend" group.

proxy_redirect off;: This line disables any automatic rewriting of HTTP redirects.

This custom Nginx configuration enables load-balancing across multiple instances of our Node.js application, ensuring better performance, high availability, and efficient utilization of resources. With this configuration, Nginx acts as a reverse proxy, directing incoming requests to one of the backend servers in the "backend" group, effectively distributing the load and improving overall application responsiveness.

Our folder and file structure should now look like this :

Now it is time to build our docker image. Since we are still on VS-code, we will start by building the docker-compose.dev.yml file. afterward, when we deploy our Virtual machine using Azure or any other third-party cloud service of your choice, we will then run the docker-compose.prod.yml file.

To build our Docker image and work with Docker Compose, you will need to have Docker and Docker Compose installed on your machine. You can follow the links below to find the installation instructions that work best for your operating system:

Docker Installation:
- For Windows: Install Docker Desktop on Windows
- For macOS: Install Docker Desktop on Mac
- For Linux: Install Docker Engine on Linux
Docker Compose Installation:
- For all platforms: Install Docker Compose

Please choose the appropriate link for your operating system and follow the step-by-step instructions provided to install Docker and Docker Compose. Once installed, you will be able to proceed with containerizing and deploying your applications using Docker and Docker Compose.

Let's proceed with building our container (and image, if one doesn't exist yet).

To do this, open your terminal and execute the following command:

docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --scale node-app=3 -d --build

Now, let's break down and understand this command:

docker-compose: This is the command-line tool we use to interact with Docker Compose.
-f docker-compose.yml -f docker-compose.dev.yml: We are using two Compose files here, docker-compose.yml and docker-compose.dev.yml, to define configurations for both the general compose configuration and the development environment.
up: This option tells Compose to create and start the containers.
--scale node-app=3: It scales the node-app service to run three instances, effectively setting up load balancing across these instances.
-d: The containers run in detached mode, meaning they will continue to run in the background.
--build: This flag ensures that Docker builds the image from the Dockerfile before starting the container.

By running this command, we initiate the process of creating and launching our containers based on the configurations we defined in the Compose files. The --scale option ensures that three instances of our Node.js application will be running concurrently, allowing us to efficiently handle incoming traffic and improve performance through load balancing.

If everything has been set up correctly your terminal should look like this :

Let's check the status of our running containers by executing the docker ps command. After scaling our Node.js application with three instances, we should observe three containers running.

However, there might be an issue with the Nginx service, which can be identified by running the docker-compose logs -f command. This log display is likely to reveal an error associated with the Nginx container, which is caused by the way we named our server files in the Nginx configuration (Further details on this will be explained later). The error will look like this :

To resolve this error, we need to ensure that the server names in our Nginx configuration file match the servers created by our containers. After making these adjustments, we can rebuild our containers using the following command:

docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --scale node-app=3 -d --build
By running this updated build command, all our containers, including Nginx, will be up and running without any issues.

As a reminder, we have made changes to our Nginx configuration file to ensure it matches the expected service names that Nginx requires. The updated Nginx configuration file should now look like this:

./nginx/default.conf

upstream backend { # this is the name of the upstream block
    server learningdocker-node-app-1:3000;
    server learningdocker-node-app-2:3000;
    server learningdocker-node-app-3:3000;
}

server {
    listen 80; # this is the port that the server will listen on

    location /api/ {
        proxy_set_header X-Real-IP $remote_addr; # this is required to pass on the client's IP to the node app
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # this is required to pass on the client's IP to the node app
        proxy_set_header Host $http_host; # this is required to pass on the client's IP to the node app
        proxy_set_header X-NginX-Proxy true; # this is required to pass on the client's IP to the node app
        proxy_pass http://backend; # this is the name of the upstream block
        proxy_redirect off; # this is required to pass on the client's IP to the node app
    }
}

Now if we run docker ps again, we should see four running containers.

Now, let's verify if our application is effectively load-balancing API calls among the three instances of our Node-API.

To do this, I have added a console.log("testing nginx") statement in the "get all users" endpoint of our Node.js application.

We will now make multiple requests to this endpoint to observe how well Nginx distributes these requests among the instances that have been created.

By running the load-balanced setup, we can assess the even distribution of API calls and ensure that our system is effectively utilizing the resources provided by the three instances. This testing will help us validate that Nginx is indeed handling load balancing as expected, improving the overall performance and scalability of our application.

Don't forget that we are working with sessions, so we must log in again before we can access the get-all user's endpoint.

GET ALL USERS:

In the "get-all-users" endpoint, I have triggered an API call 8 times consecutively to simulate multiple requests being made to our application.

To observe the real-time results of our experiment, I will open three separate terminal instances. In each terminal, I will run the following commands: docker logs -f learningdocker-node-app-1, docker logs -f learningdocker-node-app-2, and docker logs -f learningdocker-node-app-3. These commands will allow me to continuously follow the log outputs of each container to see how our application is load-balancing the API calls among the three instances of our Node-API.

RESULT:

The outcome of the experiment indicates that our load balancer is functioning as expected. It has effectively distributed the API requests among the Node instances that our container created. This demonstrates that our application is successfully load balancing and handling the requests in a balanced and efficient manner.

Excellent! Up to this point, our application is running smoothly in the development environment. However, to make it production-ready, we'll need to deploy it on a virtual machine. Creating a virtual machine is a straightforward process. For this tutorial, I'll demonstrate using Microsoft Azure as the cloud provider. However, keep in mind that you have the flexibility to choose any cloud provider you prefer, such as Google Cloud, AWS, UpCloud, or others. The essential requirement is to set up a Linux server, and any of these providers will be suitable for the task at hand. Let's proceed with the deployment process!

Once you're signed in, click on "Create a resource" in the top-left corner of the dashboard.

In the search bar, type "Virtual Machine" and select "Virtual Machines" from the suggested results.

Click on "Add" to create a new virtual machine.

Now, let's configure the virtual machine:

a. Basics:

Choose your subscription.
Create a new resource group or use an existing one.
Enter a unique virtual machine name.
Choose a region close to your target audience for better
performance.
Select "Ubuntu Server" as the image.

b. Instance details:

Choose a virtual machine size based on your needs (e.g., S
tandard B2s).
Enable "SSH public key" authentication and provide your public
SSH key. This allows you to sign in using SSH securely.

c. Disks:
Choose your preferred OS disk settings, usually, the default
settings are sufficient.

d. Networking:

Create a new virtual network or select an existing one.
Choose a subnet within the virtual network.
Enable "Public IP" and choose "Static" for a consistent IP
address.
Open port 22 for SSH (necessary for remote login), 80 for HTTP
and 443 for HTTPS.

e. Management:

Choose "Enable" for Boot diagnostics to troubleshoot startup
issues if necessary.

f. Advanced:

Customize any additional settings according to your
requirements.
Once you've completed the configuration, click on "Review +
create" to review your choices.

Review the details to ensure everything is correct, and then click on "Create" to start deploying your virtual machine.

Azure will now create the virtual machine based on your configuration. This process may take a few minutes.

If everything works fine, your Virtual machine should be up and running like this

After the virtual machine is successfully deployed, you can access it using SSH. To log in to the Ubuntu server, open your terminal and execute the following command:

ssh -i /path/to/your/sshkey.pem azureuser@your_external_ip

Replace /path/to/your/sshkey.pem with the path to your SSH private key file and azureuser with your SSH username. The your_external_ip should be replaced with the public IP address assigned to your virtual machine.

Once connected, your terminal prompt will look like this:

azureuser@your_virtual_machine_name:~$

Here is a visual representation :

Now you have secure access to your Ubuntu server, and you can perform various configurations and deploy your applications as needed. Remember to keep your server secure by using SSH keys and regularly updating your system packages.

Next thing we have to do now is install docker and docker compose on our newly created ubuntu server.

Now, our next step is to install Docker and Docker Compose on the Ubuntu server we just created.

To install the latest stable versions of Docker CLI, Docker Engine, and their dependencies :

# 1. download the script
#
#   $ curl -fsSL https://get.docker.com -o install-docker.sh
#
# 2. verify the script's content
#
#   $ cat install-docker.sh
#
# 3. run the script with --dry-run to verify the steps it executes
#
#   $ sh install-docker.sh --dry-run
#
# 4. run the script either as root, or using sudo to perform the installation.
#
#   $ sudo sh install-docker.sh
#

After the installation, verify the successful download of Docker by running docker -v in the terminal.

Next, we need to download Docker Compose, just as we did during development.

To download docker-compose, simply copy and paste the following command into your terminal :

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version

Now we have successfully installed docker and docker-compose.

Now, we need to set up our environment variables. Remember that our API will be listening for specific variables that we didn't commit to GitHub.

To set up the environment variables, we will create and add them to the .env file located in the root of our server. You can use the command sudo nano .env to open and edit the .env file. After making the necessary changes, press "Ctrl + X," then "Y" on your keyboard to save the changes before pressing the enter button. This will ensure that the environment variables are correctly configured and saved.

To verify if the changes you made have been updated successfully, use the command cat .env this will display the content of the .env file.

You should get something that looks like this :

REDIS_HOST= visit https://app.redislabs.com to get your redis host
REDIS_PORT= visit https://app.redislabs.com to get your redis port
REDIS_PASSWORD= visit https://app.redislabs.com to get your redis password
MONGO_USERNAME= vist https://cloud.mongodb.com to get your mongo username
MONGO_PASSWORD= visit https://cloud.mongodb.com to get your mongo password
SESSION_SECRET= use any random string
NODE_ENV= development or production

However, there is one issue to address. Currently, if we build the container, our API will be unable to read the .env file on our host machine unless we establish a way to link our application and persist the environment variables through reboots. To tackle this problem, we will edit the .profile file and add the following code at the bottom:

set -o allexport
source /home/azureuser/.env
set +o allexport

This way, our API will have access to the required environment variables, and we can keep them confidential and isolated from the codebase.

Note : /home/azureuser/.env is the path to my env file. Kindly replace yours to match the absolute path of your .env file on your host machine.

To access the profile file, ensure that you are in the root directory using the following command.

pwd

Then use sudo nano .profile command to open the profile file in a text editor.

After editing, make sure you save your file and exit the editor.

When you type cat .profile in your terminal, it should be displayed like this :

To apply the changes we made to the .profile file, you'll need to log out of your server. After logging back in, you can confirm if the .env file is now persistent and readable by the Node.js application by using the command printenv. In the output, if you find all the environment variables you added in the .env file, then everything is set up correctly. However, if some variables are missing, you should troubleshoot the issue until all your environment variables are displayed when you use the printenv command.

Now we will clone the app we developed and pushed to Github:

Simply type git clone https://github.com/REALSTEVEIG/USING_DOCKER and CD into the project. In my case, the project name will be USING_DOCKER.

Since we have already installed docker, set up our environment variable, all we need to do now is run the build command but this time around, we will build using the docker-compose.prod.yml file.

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up --scale node-app=3 -d --build

This command will build the image and create the required containers on the host machine and now we run into another error :

Upon closer inspection, you'll notice that the service names have changed due to pulling this project from GitHub. Consequently, the build process has modified the container names, which causes Nginx to be unaware of them. As a result, the server encounters issues recognizing the new container names.

To resolve this error, we need to go to our VsCode and modify the server names to match what Nginx can recognize. After making the necessary changes, we will push the updates to GitHub. On our Ubuntu server, we'll pull these changes and then run the build command again. This way, Nginx will correctly recognize the server names, and the issue will be resolved.

Our ./nginx/default.conf file should now look like this :

upstream backend { # this is the name of the upstream block
    server using_docker_node-app_1:3000;
    server using_docker_node-app_2:3000;
    server using_docker_node-app_3:3000;
}

server {
    listen 80; # this is the port that the server will listen on

    location /api/ {
        proxy_set_header X-Real-IP $remote_addr; # this is required to pass on the client's IP to the node app
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # this is required to pass on the client's IP to the node app
        proxy_set_header Host $http_host; # this is required to pass on the client's IP to the node app
        proxy_set_header X-NginX-Proxy true; # this is required to pass on the client's IP to the node app
        proxy_pass http://backend; # this is the name of the upstream block
        proxy_redirect off; # this is required to pass on the client's IP to the node app
    }
}

Now we rebuild the container on our Ubuntu server using the same command:

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up --scale node-app=3 -d --build

If we run docker ps Nginx should now be running alongside three instances of our Nodejs API.

Now let us test our API with the external IP provider by the cloud provider. In my case 20.69.20.104.

As you can observe, the Login route is functioning correctly. Now, let's verify if our Load-balancing is working as intended.

Create 8 requests, similar to what we did during development for the "get all users" endpoint, and observe if Nginx appropriately proxies our requests across the different Node instances. This will help us ensure that our Load-balancing mechanism is functioning as expected.

From the image above, we can confidently conclude that our load balancing works impeccably, efficiently distributing API requests among the different Node instances as expected.

With this, we have reached the conclusion of this comprehensive tutorial. Throughout this guide, we have covered a wide array of topics, including caching using Redis, load-balancing with Nginx, containerizing our application using Docker, and migrating our API to a Linux Ubuntu server on the Microsoft Azure cloud service. By following this tutorial, you have acquired valuable skills that can greatly enhance your application's performance, scalability, and deployment process.

As you continue your journey in the world of DevOps and cloud computing, there are endless possibilities to explore. You can dive deeper into deploying your API, attaching a custom domain, and implementing advanced load-balancing strategies. Additionally, learning about Kubernetes, a powerful container orchestration tool, can further boost your expertise in managing containerized applications at scale.

Remember, continuous learning and experimentation are vital in the ever-evolving tech landscape. Don't hesitate to explore new technologies, best practices, and industry trends to stay ahead in your journey as a skilled developer.

Thank you for embarking on this learning journey with me, and I wish you all the best in your future projects and endeavors! Happy coding!

Integrating a chatbot into your Nodejs API using Dialogflow

STEVE — Mon, 10 Jul 2023 13:23:36 +0000

INTRODUCTION
Welcome. This tutorial will guide you through the process of integrating a chatbot into your Node.js API using Dialogflow. With the rapid advancement of conversational AI, chatbots have become an essential component for providing interactive and personalized experiences to users. By integrating a chatbot into your Node.js API, you can enhance the functionality and engagement of your application, allowing users to interact seamlessly with automated conversational agents.

Dialogflow, powered by Google Cloud, is a powerful natural language processing (NLP) platform that enables developers to build intelligent chatbots and virtual assistants. It offers a wide range of features, including language understanding, context management, and intent recognition, making it an ideal choice for creating conversational interfaces. By combining Dialogflow with your Node.js API, you can leverage its capabilities to understand user queries, provide accurate responses, and deliver a smooth conversational experience.

Throughout this tutorial, we will explore the step-by-step process of integrating Dialogflow into your Node.js API. We will cover the necessary setup, configuration, and implementation steps, allowing you to seamlessly connect your chatbot with your API endpoints. Whether you are building a customer support system, an e-commerce platform, or any other application that requires interactive communication, this tutorial will equip you with the knowledge and tools to integrate a chatbot effectively.

By the end of this tutorial, you will have a comprehensive understanding of how to integrate Dialogflow into your Node.js API, enabling your application to handle user queries, provide intelligent responses, and offer a more engaging and dynamic user experience. So, let's dive in and embark on this exciting journey.

CREATING AN AGENT
Before we use Dialogflow, we need to create an agent. So visit Dialog flow to get started.

Next, create an agent and assign a name to it e.g. my-first-chat-bot.

In the menu, that shows up, find and click on the gear icon - (Settings).

Under the general menu, scroll down to the project ID field and click on your project ID.

This action will redirect you to the google cloud console at Google-cloud.

Next, navigate to the IAM & Admin menu and locate the service accounts sub-menu. If you haven't registered a service account yet, you will need to create a new one.

To create a new service account, enter a suitable name for your account. By default, a service ID will be automatically generated for you, but you have the option to modify it if needed. Once you have entered the necessary details, click on the "create" button to proceed. After creating the service account, the next step is to assign a role to it. This step is crucial for granting the necessary permissions. In the role selection process, search for "Dialogflow API client" under the Dialogflow section, and select it. Once you have selected the appropriate role, click on "continue" to proceed. Finally, click on "done" to complete the process.

Please note that the images provided in this tutorial serve as a visual reference to assist you in locating the correct menu and sub-menu options. The actual appearance and arrangement of the interface may vary based on the version or configuration of the platform you are using.

Once you have successfully created a service account, the next step is to obtain your configuration keys for further integration.

To download the configuration keys, click on the "Manage Keys" option. From the dropdown menu, select "Create new key". Choose the JSON format and click on the "create" button. This action will initiate the download of a JSON file containing all the necessary configuration properties required for the integration process.

It is crucial to note that the downloaded JSON file serves as your configuration file, holding essential information needed for successful integration. Therefore, it is vital to keep this file secure and not share its contents with anyone. Safeguarding this sensitive data ensures the integrity and security of your chatbot integration.

Great! now let's jump into our code.

Create a package.json file and initialize our Nodejs project by using this command :

npm init -y

Install the following packages

npm i express dotenv uuid dialogflow

import the configuration file you downloaded earlier into your project. Now your project setup should look like this :

Next, we set up our server and import the necessary dependencies.

const dotenv = require("dotenv").config()

const express = require("express")

const dialogflow = require("dialogflow")

const uuid = require('uuid');

const PORT = process.env.PORT || 7000


const projectId = process.env.PROJECT_ID || "small-talk-2-2-2-2"
const credentialsPath = process.env.CREDENTIALS_PATH || "./small-talk-2-2-2-2-5b3b3b3b3b3b.json"

process.env.GOOGLE_APPLICATION_CREDENTIALS = credentialsPath

const start = async () => {
    try {
        app.listen(PORT, () => {
            console.log(`Server has been started on port ${PORT}`)
        })
    } catch (error) {
      console.log(error)  
    }
}

start()

Now, lets explain what is happening here :

The projectId variable is being set to either the value of the process.env.PROJECT_ID environment variable or "small-talk-2-2-2-2" if the variable is not defined.

The credentialsPath variable is being set to either the value of the process.env.CREDENTIALS_PATH environment variable or "./small-talk-2-2-2-2-5b3b3b3b3b3b.json" if the variable is not defined.

The GOOGLE_APPLICATION_CREDENTIALS environment variable is being set to the value of credentialsPath. This variable is used to specify the path to the Google Cloud service account credentials JSON file that we downloaded and imported.

Now since we are loading the credentials from our .env file, it is important to copy and paste the credentials from our JSON file into the .env file and ensure to spread all the credential properties across one variable.

Spreading the credential properties across one variable, such as CREDENTIALS, in the .env file ensures that the credentials are assigned to a single environment variable. This approach simplifies the code and enhances readability. It also allows the application to access the credentials easily by referencing the CREDENTIALS environment variable, rather than handling each credential property individually.

Your .env file should now look like this :

Now, we will establish a connection to Dialogflow by creating a function and configuring a route to handle incoming requests and responses.

async function runSample() {
  // A unique identifier for the given session
  const sessionId = uuid.v4();

  // Create a new session
  const sessionClient = new dialogflow.SessionsClient();
  const sessionPath = sessionClient.sessionPath(projectId, sessionId);

  // The text query request.
  const request = {
    session: sessionPath,
    queryInput: {
      text: {
        // The query to send to the dialogflow agent
        text: 'Who are you?',
        // The language used by the client (en-US)
        languageCode: 'en-US',
      },
    },
  };

  // Send request and log result
  const responses = await sessionClient.detectIntent(request);
  const result = responses[0].queryResult.fulfillmentText;
  const queryText = responses[0].queryResult.queryText;

  if (result) {
        return {
            user: queryText,
            bot: result
        }
} else {
    return Error("No intent matched")
  }
}

const app = express()

app.get("/", async (req, res) => {
    try {
        const result = await runSample()
        return res.status(200).json({message: "Success", result})
    } catch (error) {
        console.log(error)
        return res.status(500).json({message: "Server error", error})
    }
})

In the code above, the following actions are taking place:

An asynchronous function named runSample() is defined. It performs the following tasks:

Generates a unique identifier, sessionId, for the session.
Creates a new session using dialogflow.SessionsClient().
Constructs the session path using the projectId and sessionId.
Defines a text query request containing the query to send to the
Dialogflow agent and the language code.
Sends the request to Dialogflow using sessionClient.detectIntent() and awaits the response.
Extracts the fulfillment text and query text from the response.
Returns an object with the user query and the bot's response, if a fulfillment text is present. Otherwise, it returns an error indicating that no intent matched the query.
An Express.js application is created using express() and stored in the app variable.
A route is set up for the root URL ("/") using app.get(). The route is configured as an asynchronous function that performs the following tasks:
Calls the runSample()
Returns a JSON response with a success message and the result of the runSample() function if successful.
Logs any errors that occur during the process and returns a JSON response with an error message if there's a server error.

Overall, this code sets up a route in an Express.js application that connects to Dialogflow, sends a query, and retrieves a response. The result is then returned as a JSON response to the client. Any errors that occur during the process are logged and returned as error messages.

Our app.js file should now look like this :

const dotenv = require("dotenv").config()

const express = require("express")

const dialogflow = require("dialogflow")

const uuid = require('uuid');

const PORT = process.env.PORT || 7000

const projectId = process.env.PROJECT_ID || "small-talk-2-2-2-2"
const credentialsPath = process.env.CREDENTIALS_PATH || "./small-talk-2-2-2-2-5b3b3b3b3b3b.json"

process.env.GOOGLE_APPLICATION_CREDENTIALS = credentialsPath

async function runSample() {
  // A unique identifier for the given session
  const sessionId = uuid.v4();

  // Create a new session
  const sessionClient = new dialogflow.SessionsClient();
  const sessionPath = sessionClient.sessionPath(projectId, sessionId);

  // The text query request.
  const request = {
    session: sessionPath,
    queryInput: {
      text: {
        // The query to send to the dialogflow agent
        text: 'Who are you?',
        // The language used by the client (en-US)
        languageCode: 'en-US',
      },
    },
  };

  // Send request and log result
  const responses = await sessionClient.detectIntent(request);
  const result = responses[0].queryResult.fulfillmentText;
  const queryText = responses[0].queryResult.queryText;

  if (result) {
        return {
            user: queryText,
            bot: result
        }
} else {
    return Error("No intent matched")
  }
}

const app = express()

app.get("/", async (req, res) => {
    try {
        const result = await runSample()
        return res.status(200).json({message: "Success", result})
    } catch (error) {
        console.log(error)
        return res.status(500).json({message: "Server error", error})
    }
})

const start = async () => {
    try {
        app.listen(PORT, () => {
            console.log(`Server has been started on port ${PORT}`)
        })
    } catch (error) {
      console.log(error)  
    }
}

start()

Now let's start our server and test. If everything was set up correctly, we will get this response :

The response is a JSON object with two key-value pairs:

"message": "Success" - This indicates that the request to Dialogflow was successful and a response was received.

"result": This key holds another JSON object with two key-value pairs:

"user": "Who are you?" - This represents the user's query or input sent to Dialogflow. In this case, the user asked, "Who are you?"
"bot": "You can call me Sofia. How can I help you today?" - This is the response generated by Dialogflow's natural language processing. It indicates that the bot's name is Sofia and asks how it can assist the user.

Overall, the response demonstrates a successful interaction with Dialogflow, where the user's query is processed, and the bot responds with an appropriate answer.

At this point, our API is fully functional and ready to be integrated with a user interface. However, to optimize the performance of our chatbot, continuous training is essential. This involves adding new intents and responses in the Dialogflow dashboard, allowing the chatbot to handle a wider range of queries and interactions.

let us change the user request and see what response we will receive from the agent.

Request

Response

What happens when the user asks a question which our agent does not understand?
Well, there are predefined fallback responses like this :

Dialogflow offers extensive possibilities for customization and integration. We have the flexibility to incorporate the chatbot into various platforms such as Facebook, WhatsApp, and more. By leveraging these integrations, we can reach users across different channels and provide them with personalized conversational experiences.

Additionally, Dialogflow supports multi-language capabilities, enabling us to serve users in their preferred languages. Activating multi-language support ensures that our chatbot can communicate effectively with users from diverse linguistic backgrounds.

It's crucial to emphasize that the effectiveness of our chatbot relies on the quality of information we provide. Regularly updating and refining the training data, intents, and responses helps our agent improve its understanding and accuracy in delivering meaningful and helpful interactions.

In summary, by continuously training our agent and exploring the extensive features of Dialogflow, we can unlock a multitude of possibilities to enhance the functionality and reach of our chatbot. Taking advantage of different integration options, such as social media platforms and multi-language support, allows us to create engaging user experiences and effectively address the needs of a diverse user base. Remember, the success of our chatbot lies in the knowledge and information we feed it, so ongoing improvement and adaptation are key.

GETTING STARTED WITH CACHING: USING REDIS AND TYPESCRIPT

STEVE — Tue, 04 Jul 2023 18:18:34 +0000

INTRODUCTION:

In this Tutorial, We are going to learn the fundamentals of caching and how to implement them using Redis and Typescript/Nodejs. But before we begin, let's start with the basics.

What is Caching?
Caching is a technique used in computer systems and software applications to store and retrieve data quickly. It involves storing a copy of frequently accessed or expensive-to-compute data in a temporary storage location, called a cache so that future requests for the same data can be served faster.

The purpose of caching is to improve the performance and efficiency of a system by reducing the time and resources required to fetch data from its original source. Instead of retrieving the data from the original location, which may involve time-consuming operations like disk access or network communication, the data is retrieved from the cache, which is typically located closer to the requester and provides faster access.

PREREQUISITES
To follow along with this tutorial, you will need to have a basic understanding of NodeJS and Typescript. I will try my best to explain every required step and I am confident that I can provide thorough explanations to assist you throughout the process.

Why Redis?
When it comes to caching, Redis stands out as an exceptional choice for several reasons. Its features and capabilities make it an ideal solution for optimizing performance and improving overall system efficiency.

Redis is renowned for its incredible speed and performance. By storing data in memory, Redis enables lightning-fast data access and retrieval, making it suitable for applications that require real-time data processing and high throughput. With the ability to handle millions of requests per second and provide low-latency response times, Redis excels in scenarios where speed is of utmost importance.

By leveraging Redis as a cache, applications can store frequently accessed data in memory, eliminating the need for repetitive and resource-intensive operations. This results in improved performance, reduced response times, and a more seamless user experience.

In this tutorial, we will employ Redis for caching data retrieved from an external API with a noticeably slow response time. We will store this data in our cache and utilize it for subsequent requests made to the API. However, we will also address the scenario when the data from the original API undergoes changes. We will implement a mechanism to ensure that our cache consistently provides up-to-date data from the API, ensuring that our system remains synchronized with the latest information.

Let's get started.

First, let's initialize our Nodejs project and install the required dependencies.

npm init -y

npm i express axios cors dotenv redis

Now install the type definitions :

npm i @types/express @types/axios @types/cors @types/dotenv @types/redis

Having a tsconfig.json file ensures consistency in the compilation process across different environments and allows for easy project configuration and maintenance.

So go ahead and create one using :

tsc --init

Then configure your outDir and rootDir options to match the following structure.

Don't forget to watch for changes in your Typescript code by using tsc -w in a dedicated terminal.

Next, let us connect to Redis. To do this, you have two options.

Connect to Redis-Local-Server using localhost.
Connect to the Redis-cloud-console by creating an instance via Redis-cloud

In this tutorial, we will go with option 2. (If you don't have an account already, Kindly create one and create a database instance).

Now let's connect to our database. Our ./src/config.ts file should now have the following lines of code :

import { createClient } from 'redis';
import dotenv from 'dotenv';

dotenv.config();

export const client = createClient({
    password: process.env.REDIS_PASSWORD,
    socket: {
        host: process.env.REDIS_HOST,
        port: parseInt(process.env.REDIS_PORT || '6379', 10)
    }
});

The code above sets up a Redis client using the redis package, loads environment variables from a .env file using dotenv, and exports the Redis client for use in other parts of the codebase. The environment variables are used to configure the Redis connection, including the host, port, and password. (These variables are available on your database instance once you create an account here : Redis-cloud )

Next, Let's import the client variable in our app.ts file and connect to the database.

Configure your app.ts file to look like this :

import dotenv from "dotenv"
import express, { Request, Response } from "express"
import axios from "axios"
import { client } from "./config/connect"
import cors from "cors"

dotenv.config()

const app : express.Application = express()

const PORT = process.env.PORT || 7000

app.use(cors())
app.use(express.json())

const start = async () => {
    try {
        await client.connect()
        app.listen(PORT, () => {
            console.log(`Server is connected to redis and is listening on port ${PORT}`)
        })
    } catch (error) {
        console.log(error)
    }
}

start()

The code above imports all the installed dependencies we will need for this project, sets up an Express server, configures it to handle JSON requests, enables CORS, connects to the Redis client, and starts the server to listen for incoming requests if the connection is successful.

Now for this tutorial, I have created and deployed an API whose response is intentionally delayed by 5secs to demonstrate how caching can be useful in a real-world scenario.

So in our app.ts file, we will create two functions that make API calls like this :

async function isDataModified () {
    const response = await axios.get("https://pleasant-newt-girdle.cyclic.app/api/modified")
    return response.data.modified
}

async function getAllUsers () {
    const response = await axios.get("https://pleasant-newt-girdle.cyclic.app/api/users")
    return response.data
}

Now, let's examine each of these functions separately. The first function sends a request to an endpoint that interacts with its database. It retrieves a boolean result, which evaluates to true if there have been any recent changes. These changes encompass scenarios such as the addition of a new item POST, modification of an existing item PUT, or deletion of an item DELETE. In the absence of any changes, the result will be false.

The second function, on the other hand, straightforwardly retrieves a list of all items (in this case, users) stored in the database of that particular API.

Now, let's understand the rationale behind this approach. Why are we making two requests to an API? Remember when we mentioned the need to cache only recent information? Exactly! So, what happens when a user decides to update specific details on their profile? In such cases, we also need to update the cache, right? Absolutely correct. That's precisely what the first function accomplishes. Before serving our response, we verify if any changes have been made in the API's database to ensure that our cache is up-to-date.

Now let's create an endpoint in this API to store and retrieve information from our cache. Update the app.ts file with this endpoint.

app.use("/get-users", async (req : Request, res : Response) => {

    let result;
    let isCahed;

    try {

        const data = await isDataModified()

        if (data === true) {
            result = await getAllUsers()
            isCahed = false
            await client.set("all_users", JSON.stringify(result))
        } 

        else {

            const isCahedInRedis = await client.get("all_users");

            if (isCahedInRedis) {

                isCahed = true
                result = JSON.parse(isCahedInRedis)
            }

           else {
                result = await getAllUsers()
                isCahed = false

                await client.set("all_users", JSON.stringify(result))
           }

        }

        return res.status(200).json({
            isCahed,
            result : result
        })
    } catch (error) {
        console.log(error)
        return res.status(500).json({error})
    }
})

Now, let's explain what is happening here step by step:

Inside the route handler function:

a. Two variables, result and isCached, are created to store the API request result and caching status.

b. The isDataModified() function is called to check if there have been any modifications in the database. The result is stored in the data variable.

c. If modifications are detected (when data is true), it means the cache needs to be updated. The getAllUsers() function is called to retrieve all user data from the API. The result is assigned to the result variable, and the isCached variable is set to false. The retrieved data is then stored in the Redis cache using the client.set() method.

d. If no modifications are detected, it means the cached data is still valid. The code checks if the data is already cached in Redis using the client.get() method. If cached data exists, it is assigned to the result variable, and the isCached variable is set to true.

e. If no cached data exists, the getAllUsers() function is called to retrieve the user data from the API. The result is assigned to the result variable, and the isCached variable is set to false. The retrieved data is then stored in the Redis cache using the client.set() method.

f. Finally, the code sends a JSON response with a status of 200. The response includes the isCached status and the result data.

If any errors occur during the process, they are caught in the catch block. A JSON response with a status of 500 and the error message is returned.

To summarize, this code sets up an endpoint that fetches user data from an API. It checks if the data has been modified, updates the cache if needed, and returns the cached data or retrieves fresh data from the API.

And that pretty much does the job. Our app.ts file should now look like this :

import dotenv from "dotenv"
import express, { Request, Response } from "express"
import axios from "axios"
import { client } from "./config/connect"
import cors from "cors"

dotenv.config()

const app : express.Application = express()

const PORT = process.env.PORT || 7000

app.use(cors())
app.use(express.json())

async function isDataModified () {
    const response = await axios.get("https://pleasant-newt-girdle.cyclic.app/api/modified")
    return response.data.modified
}

async function getAllUsers () {
    const response = await axios.get("https://pleasant-newt-girdle.cyclic.app/api/users")
    return response.data
}

app.use("/get-users", async (req : Request, res : Response) => {

    let result;
    let isCached;

    try {

        const data = await isDataModified()

        if (data === true) {
            result = await getAllUsers()
            isCached = false
            await client.set("all_users", JSON.stringify(result))
        } 

        else {

            const isCachedInRedis = await client.get("all_users");

            if (isCachedInRedis) {

                isCached = true
                result = JSON.parse(isCachedInRedis)
            }

           else {
                result = await getAllUsers()
                isCached = false

                await client.set("all_users", JSON.stringify(result))
           }

        }

        return res.status(200).json({
            isCached,
            result : result
        })
    } catch (error) {
        console.log(error)
        return res.status(500).json({error})
    }
})

const start = async () => {
    try {
        await client.connect()
        app.listen(PORT, () => {
            console.log(`Server is connected to redis and is listening on port ${PORT}`)
        })
    } catch (error) {
        console.log(error)
    }
}

start()

Now run the server and let us test our endpoints :

The first thing you will notice when you hit this endpoint http://localhost:7000/get-users is how long it took to get a response. As I mentioned before, that is intentional, it is meant to simulate how real-world applications, that are CPU intensive behave when caching is not implemented.

Next, you will notice that the first property from the response, which is isCached reads false. This means that this data does not exist in our cache but has been added immediately. How can we confirm that? well, lets make the same request again by simply refreshing our browser. This is what we get :

Notice how, the response time is reduced as you keep refreshing the page? also, Notice that the isCachedproperty now reads true.

To test what will happen when we alter the database, I invite you to modify the response by creating, editing, or deleting users using any of the endpoints below :

Create user (POST): https://pleasant-newt-girdle.cyclic.app/api/user
Update user (PUT)/Delete user (DELETE): https://pleasant-newt-girdle.cyclic.app/api/user/:id

For every time, you makes these requests successfully (POST, PUT and DELETE) the isCached value becomes false, prompting a delayed response to fetch current data and update the cache. Remember, you can always refresh your endpoint at http://localhost:7000/get-users to visualize the changes happening live.

Here is an example of how to make these requests on Postman.

CREATE NEW USER

UPDATE USER

DELETE USER

Now, if for some reason, the API, https://pleasant-newt-girdle.cyclic.app/api/user ceases to exist in the future, you can also try out this similar public API https://reqres.in/api/users?delay=3. Here is how it works: The response time of the API is contingent on the query parameter delay. That means if you set the delay=3, it will take three seconds before the API returns a response. This means that you can practice the caching mechanism with this API.

But wait, how do we actually visualize the data in our cache? Well, you can download a tool like RedisInsight, connect your database instance, and voila, you can now visualize and query your cache.

Here is a link to the complete code on GitHub. Thank you for sticking around till the end.

FINAL NOTES:

Congratulations, if you got this far. By now, you should have a solid understanding of how Redis can be leveraged as a powerful caching solution. However, caching with Redis extends beyond just speeding up data retrieval. In this final note, let's explore some additional use cases and discuss the remarkable usefulness of caching.

Session Caching: Redis is an excellent choice for storing session data. By caching session information, you can achieve high-performance session management, improve user experience, and reduce database load. Redis's ability to set expiration times on keys makes it perfect for managing session timeouts and automatically cleaning up expired sessions.

Full-page Caching: Redis can be used to cache entire HTML pages, eliminating the need to regenerate them on each request. By serving cached pages directly from Redis, you can dramatically reduce the response time and alleviate the load on your application servers.

Result Caching: Redis enables you to cache the results of complex or time-consuming computations. For example, if your application involves heavy calculations or data processing, you can store the computed results in the Redis cache and retrieve them when needed, avoiding redundant computations.

Leaderboards and Counters: Redis's sorted sets and atomic increment operations make it an excellent choice for implementing leaderboards, vote counters, or popularity rankings. By caching these frequently changing metrics, you can efficiently update and display them in real-time.

Pub/Sub Messaging: Redis supports Publish/Subscribe (Pub/Sub) messaging, allowing you to build real-time communication channels, notifications, and event-driven architectures. By caching messages or maintaining subscription lists, Redis facilitates the implementation of scalable, high-performance messaging systems.

The usefulness of caching with Redis cannot be overstated. By intelligently caching data, you can achieve significant performance improvements, reduce latency, and enhance the overall scalability of your applications. However, it's crucial to consider cache invalidation and maintain data consistency when working with caching systems.

Node.js and TypeScript Tutorial: Build a rest API with Typescript, NodeJS, and a file-based storage system.

STEVE — Mon, 19 Jun 2023 20:57:52 +0000

Introduction

Welcome to my blog! In this tutorial, I will guide you through the process of building a robust micro e-commerce API using Node.js, Express, and TypeScript. Together, we will explore various features and techniques that will empower you to create a powerful API for your e-commerce applications.

One of our key decisions in this project was to implement a file-based storage system instead of relying on traditional databases like MongoDB. This approach offers simplicity and ease of implementation, making it ideal for smaller-scale applications or scenarios where a full-fledged database management system may be unnecessary.

The tutorial will cover essential topics such as user management, product handling, and authentication.

You'll gain hands-on experience working with features that span both user and product data, demonstrating how these entities interact within an e-commerce API. By the end of this tutorial, you'll have a comprehensive understanding of building a powerful API that enables seamless interactions with user and product resources.

So, join me on this exciting journey as we dive into creating a micro e-commerce API using Node.js, Express, and TypeScript.

Get Started with TypeScript in Node.js
Start by creating a project directory that looks like this.

Next, initialize a Node.js project within the project directory by creating a package.json file with default settings, using this command :

npm init -y

Install Project Dependencies

Your Node.js project requires a couple of dependencies to create a secure Express server with TypeScript. Install them like so:

npm i express dotenv helmet cors http-status-codes uuid bcryptjs
To use TypeScript, you also need to install a stable version of typescript as a developer dependency:

npm i -D typescript

To use TypeScript effectively, you need to install type definitions for the packages you installed previously:

npm i -D @types/express @types/dotenv @types/helmet @types/cors @types/http-status-codes @types/uuid @types/bcryptjs

Populate the .env hidden file with the following variable that defines the port your server can use to listen for requests:

PORT=7000

Next, locate the app.js file in the root of the src folder and import the project dependencies you installed earlier and load any environmental variables from the local .env file using the dotenv.config() method:

import express from "express"
import * as dotevnv from "dotenv"
import cors from "cors"
import helmet from "helmet"

dotevnv.config()

if (!process.env.PORT) {
    console.log(`No port value specified...`)
}

const PORT = parseInt(process.env.PORT as string, 10)

const app = express()

app.use(express.json())
app.use(express.urlencoded({extended : true}))
app.use(cors())
app.use(helmet())

app.listen(PORT, () => {
    console.log(`Server is listening on port ${PORT}`)
})

In this code snippet, a Node.js application is being set up using the Express framework. Here's a breakdown of what's happening:

The required modules are imported:

express is imported as the main framework for building the web application.

dotenv is imported to handle environment variables.

cors is imported to enable Cross-Origin Resource Sharing.

helmet is imported to add security headers to HTTP responses.

The code checks if the PORT environment variable is defined. If not, a message is logged to the console.

The PORT variable is parsed from a string to an integer using parseInt().

An instance of the Express application is created using express() and assigned to the app variable.

Middleware functions are added to the Express application:

express.json() is used to parse JSON bodies of incoming requests.

express.urlencoded({extended : true}) is used to parse URL-encoded bodies of incoming requests.

cors() is used to enable Cross-Origin Resource Sharing.

helmet() is used to enhance the security of the application by setting various HTTP headers.

The Express application starts listening on the specified PORT by calling app.listen(). Once the server is running, a message indicating the port number is logged to the console.

Improve TypeScript Development Workflow

The TypeScript compilation process can increase the bootstrapping time of an application. However, you don't need to recompile the entire project whenever there's a change in its source code. You can set up ts-node-dev to significantly decrease the time it takes to restart your application when you make a change.

Start by installing this package to power up your development workflow:

npm i -D ts-node-dev

ts-node-dev restarts a target Node.js process when any of the required files change. However, it shares the Typescript compilation process between restarts, which can significantly increase the restart speed.

You can create a dev npm script in package.json to run your server. Update your package.json file like this.

{ "name": "typescript-nodejs", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "dev": "ts-node-dev --pretty --respawn ./src/app.ts" }, "keywords": [], "author": "", "license": "ISC", "dependencies": { "@types/nanoid": "^3.0.0", "@types/uuid": "^9.0.2", "bcryptjs": "^2.4.3", "cors": "^2.8.5", "dotenv": "^16.3.0", "express": "^4.18.2", "helmet": "^7.0.0", "http-status-codes": "^2.2.0", "nanoid": "^4.0.2", "uuid": "^9.0.0" }, "devDependencies": { "@types/bcryptjs": "^2.4.2", "@types/cors": "^2.8.13", "@types/dotenv": "^8.2.0", "@types/express": "^4.17.17", "@types/helmet": "^4.0.0", "@types/http-status-codes": "^1.2.0", "ts-node-dev": "^2.0.0" } }

Let's briefly break down the options that ts-node-dev takes:

--respawn: Keep watching for changes after the script has exited.

--pretty: Use pretty diagnostic formatter (TS_NODE_PRETTY).

./src/app.ts: This is the application's entry file.

Now, simply run the dev script to launch your project:

npm run dev

If everything is working correctly, you'll see a message indicating that the server is listening for requests on port 7000.

Model Data with TypeScript Interfaces

Before creating any routes, define the structure of the data you want to manage. Our user database will have the following properties:

id : (string) Unique identifier for the item record.
username : (string) Name of the item.
email : (number) Price of the item in cents.
password : (string) Description of the item.

Populate src/users/user.interface.ts with the following definition:

export interface User {
    username : string,
    email : string,
    password : string
}

export interface UnitUser extends User {
    id : string
}

export interface Users {
    [key : string] : UnitUser
}

This code defines three TypeScript interfaces:

The User interface represents a basic user object with three properties:

username, which is a string representing the username of the user.
email, which is a string representing the email address of the user.
password, which is a string representing the password of the user.

The UnitUser interface extends the User interface and adds an id property:

id, which is a string representing the unique identifier of the user.

The Users interface represents a collection of user objects with dynamic keys:

[key: string] indicates that the keys of the Users object can be any string.
The values of the Users object are of type UnitUser, which means each user object in the collection should conform to the UnitUser interface.
In simpler terms, these interfaces define the structure and types of user objects. The User interface defines the basic properties of a user, while the UnitUser interface adds an id property to represent a user with a unique identifier. The Users interface represents a collection of user objects, where the keys are strings and the values are UnitUser objects.

Next, we will create the logic for our data storage. you can call it a database if you like.
Populate src/users/user.database.ts with the following code:

import { User, UnitUser, Users } from "./user.interface";
import bcrypt from "bcryptjs"
import {v4 as random} from "uuid"
import fs from "fs"

let users: Users = loadUsers() 

function loadUsers () : Users {
  try {
    const data = fs.readFileSync("./users.json", "utf-8")
    return JSON.parse(data)
  } catch (error) {
    console.log(`Error ${error}`)
    return {}
  }
}

function saveUsers () {
  try {
    fs.writeFileSync("./users.json", JSON.stringify(users), "utf-8")
    console.log(`User saved successfully!`)
  } catch (error) {
    console.log(`Error : ${error}`)
  }
}

export const findAll = async (): Promise<UnitUser[]> => Object.values(users);

export const findOne = async (id: string): Promise<UnitUser> => users[id];

export const create = async (userData: UnitUser): Promise<UnitUser | null> => {

  let id = random()

  let check_user = await findOne(id);

  while (check_user) {
    id = random()
    check_user = await findOne(id)
  }

  const salt = await bcrypt.genSalt(10);

  const hashedPassword = await bcrypt.hash(userData.password, salt);

  const user : UnitUser = {
    id : id,
    username : userData.username,
    email : userData.email,
    password: hashedPassword
  };

  users[id] = user;

  saveUsers()

  return user;
};

export const findByEmail = async (user_email: string): Promise<null | UnitUser> => {

  const allUsers = await findAll();

  const getUser = allUsers.find(result => user_email === result.email);

  if (!getUser) {
    return null;
  }

  return getUser;
};

export const comparePassword  = async (email : string, supplied_password : string) : Promise<null | UnitUser> => {

    const user = await findByEmail(email)

    const decryptPassword = await bcrypt.compare(supplied_password, user!.password)

    if (!decryptPassword) {
        return null
    }

    return user
}

export const update = async (id : string, updateValues : User) : Promise<UnitUser | null> => {

    const userExists = await findOne(id)

    if (!userExists) {
        return null
    }

    if(updateValues.password) {
        const salt = await bcrypt.genSalt(10)
        const newPass = await bcrypt.hash(updateValues.password, salt)

        updateValues.password = newPass
    }

    users[id] = {
        ...userExists,
        ...updateValues
    }

    saveUsers()

    return users[id]
}

export const remove = async (id : string) : Promise<null | void> => {

    const user = await findOne(id)

    if (!user) {
        return null
    }

    delete users[id]

    saveUsers()
}

Let me explain every function in the code above :

loadUsers: This function reads the data from a file called "users.json" using the fs module. It attempts to parse the data as JSON and returns it as the users object. If an error occurs during the process, it logs the error and returns an empty object.

saveUsers: This function saves the users object to the "users.json" file by writing the JSON string representation of the users object using the fs module's writeFileSync method. If an error occurs during the process, it logs the error.

findAll: This function returns a promise that resolves to an array of UnitUser objects. It uses Object.values(users) to extract the values (users) from the users object.

findOne: This function takes an id parameter and returns a promise that resolves to the UnitUser object corresponding to that id in the users object.

create: This function takes a userData object as input and returns a promise that resolves to the newly created UnitUser object. It generates a random id using the uuid package and checks if a user with that id already exists. If a user with that id exists, it generates a new id until a unique one is found. It then hashes the userData object's password using bcrypt and saves the hashed password in the UnitUser object. The UnitUser object is added to the users object, saved using saveUsers, and returned.

findByEmail: This function takes a user_email parameter and returns a promise that resolves to a UnitUser object if a user with the specified email exists, or null otherwise. It retrieves all users using findAll and finds the user with the matching email using the find method.

comparePassword: This function takes an email and supplied_password as parameters and returns a promise that resolves to a UnitUser object if the supplied password matches the user's stored password, or null otherwise. It calls findByEmail to retrieve the user by email and then uses bcrypt.compare to compare the hashed stored password with the supplied password.

update: This function takes an id and updateValues as parameters and returns a promise that resolves to the updated UnitUser object if the user with the specified id exists. It checks if the user exists using findOne and updates the user's password if updateValues contains a new password. The user's properties are updated with the values from updateValues, and the users object is saved using saveUsers.

remove: This function takes an id parameter and returns a promise that resolves to null if the user with the specified id doesn't exist, or void otherwise. It uses findOne to check if the user exists and deletes the user from the users object using the delete keyword. The updated users object is then saved using saveUsers.

These functions serve as the methods our API can use to process and retrieve information from the database.

Next, let all import all the required functions and modules into the routes file ./src/users.routes.ts and populate as follows :

import express, {Request, Response} from "express"
import { UnitUser, User } from "./user.interface"
import {StatusCodes} from "http-status-codes"
import * as database from "./user.database"

export const userRouter = express.Router()

userRouter.get("/users", async (req : Request, res : Response) => {
    try {
        const allUsers : UnitUser[] = await database.findAll()

        if (!allUsers) {
            return res.status(StatusCodes.NOT_FOUND).json({msg : `No users at this time..`})
        }

        return res.status(StatusCodes.OK).json({total_user : allUsers.length, allUsers})
    } catch (error) {
        return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error})
    }
})

userRouter.get("/user/:id", async (req : Request, res : Response) => {
    try {
        const user : UnitUser = await database.findOne(req.params.id)

        if (!user) {
            return res.status(StatusCodes.NOT_FOUND).json({error : `User not found!`})
        }

        return res.status(StatusCodes.OK).json({user})
    } catch (error) {
        return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error})
    }
})

userRouter.post("/register", async (req : Request, res : Response) => {
    try {
        const { username, email, password } = req.body

        if (!username || !email || !password) {
            return res.status(StatusCodes.BAD_REQUEST).json({error : `Please provide all the required parameters..`})
        }

        const user = await database.findByEmail(email) 

        if (user) {
            return res.status(StatusCodes.BAD_REQUEST).json({error : `This email has already been registered..`})
        }

        const newUser = await database.create(req.body)

        return res.status(StatusCodes.CREATED).json({newUser})

    } catch (error) {
        return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error})
    }
})

userRouter.post("/login", async (req : Request, res : Response) => {
    try {
        const {email, password} = req.body

        if (!email || !password) {
            return res.status(StatusCodes.BAD_REQUEST).json({error : "Please provide all the required parameters.."})
        }

        const user = await database.findByEmail(email)

        if (!user) {
            return res.status(StatusCodes.NOT_FOUND).json({error : "No user exists with the email provided.."})
        }

        const comparePassword = await database.comparePassword(email, password)

        if (!comparePassword) {
            return res.status(StatusCodes.BAD_REQUEST).json({error : `Incorrect Password!`})
        }

        return res.status(StatusCodes.OK).json({user})

    } catch (error) {
        return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error})
    }
})


userRouter.put('/user/:id', async (req : Request, res : Response) => {

    try {

        const {username, email, password} = req.body

        const getUser = await database.findOne(req.params.id)

        if (!username || !email || !password) {
            return res.status(401).json({error : `Please provide all the required parameters..`})
        }

        if (!getUser) {
            return res.status(404).json({error : `No user with id ${req.params.id}`})
        }

        const updateUser = await database.update((req.params.id), req.body)

        return res.status(201).json({updateUser})
    } catch (error) {
        console.log(error) 
        return res.status(500).json({error})
    }
})

userRouter.delete("/user/:id", async (req : Request, res : Response) => {
    try {
        const id = (req.params.id)

        const user = await database.findOne(id)

        if (!user) {
            return res.status(StatusCodes.NOT_FOUND).json({error : `User does not exist`})
        }

        await database.remove(id)

        return res.status(StatusCodes.OK).json({msg : "User deleted"})
    } catch (error) {
        return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error})
    }
})

This is what each function does :

userRouter.get("/users"): This function handles a GET request to "/users". It calls the findAll function from the database module to retrieve all users. If no users are found, it returns a 404 status code with a message. If users are found, it returns a 200 status code with the total number of users and the array of all users.

userRouter.get("/user/:id"): This function handles a GET request to "/user/:id" where :id represents a specific user's ID. It calls the findOne function from the database module to retrieve the user with the specified ID. If the user is not found, it returns a 404 status code with an error message. If the user is found, it returns a 200 status code with the user object.

userRouter.post("/register"): This function handles a POST request to "/register" for user registration. It extracts the username, email, and password from the request body. If any of these fields are missing, it returns a 400 status code with an error message. It calls the findByEmail function from the database module to check if the email is already registered. If the email is found, it returns a 400 status code with an error message. If the email is not found, it calls the create function from the database module to create a new user and returns a 201 status code with the newly created user object.

userRouter.post("/login"): This function handles a POST request to "/login" for user login. It extracts the email and password from the request body. If any of these fields are missing, it returns a 400 status code with an error message. It calls the findByEmail function from the database module to check if the email exists. If the email is not found, it returns a 404 status code with an error message. If the email is found, it calls the comparePassword function from the database module to check if the supplied password matches the stored password. If the passwords don't match, it returns a 400 status code with an error message. If the passwords match, it returns a 200 status code with the user object.

userRouter.put('/user/:id'): This function handles a PUT request to "/user/:id" where :id represents a specific user's ID. It extracts the username, email, and password from the request body. If any of these fields are missing, it returns a 401 status code with an error message. It calls the findOne function from the database module to check if the user with the specified ID exists. If the user is not found, it returns a 404 status code with an error message. If the user is found, it calls the update function from the database module to update the user's details and returns a 201 status code with the updated user object.

userRouter.delete("/user/:id"): This function handles a DELETE request to "/user/:id" where :id represents a specific user's ID. It extracts the id from the request parameters. It calls the findOne function from the database module to check if the user with the specified ID exists. If the user is not found, it returns a 404 status code with an error message. If the user is found, it calls the remove function from the database module to delete the user and returns a 200 status code with a success message.

All These functions define the routes and corresponding logic for user-related operations such as retrieving all users, retrieving a specific user, registering a new user, logging in a user, updating a user's details, and deleting a user.

finally, to make API calls to these routes we need to import them into our app.ts file and update our code like this :

import express from "express"
import * as dotevnv from "dotenv"
import cors from "cors"
import helmet from "helmet"
import { userRouter } from "./users/users.routes"

dotevnv.config()

if (!process.env.PORT) {
    console.log(`No port value specified...`)
}

const PORT = parseInt(process.env.PORT as string, 10)

const app = express()

app.use(express.json())
app.use(express.urlencoded({extended : true}))
app.use(cors())
app.use(helmet())

app.use('/', userRouter)

app.listen(PORT, () => {
    console.log(`Server is listening on port ${PORT}`)
})

Great! now let's start our server and test our API using Postman.

run npm run dev in your terminal

your terminal should be similar to this

[INFO] 20:55:40 ts-node-dev ver. 2.0.0 (using ts-node ver. 10.9.1, typescript ver. 5.1.3) Server is listening on port 7000

Great! Let's make calls to our endpoints.

Register users

Login users

Get all users

Get a single user

Update user

Delete user :

NOTE : If you have added users, your users.json file should continuously append new users and should look like this.

Users-data-storage-file :

Finally, let us create the login and routes for our products.
So let's duplicate the contents of our users interface with minor changes into the file ./src/product.interface.ts

export interface Product {
    name : string,
    price : number;
    quantity : number;
    image : string;
}

export interface UnitProduct extends Product {
    id : string
}

export interface Products {
    [key : string] : UnitProduct
}

You can reference the section on the Users interface for details about what these interfaces do.

Next, just like in the ./src/users.database.ts file, let us populate the ./src/products.database.ts with a similar logic.

import { Product, Products, UnitProduct } from "./product.interface";
import { v4 as random } from "uuid";
import fs from "fs";

let products: Products = loadProducts();

function loadProducts(): Products {
  try {
    const data = fs.readFileSync("./products.json", "utf-8");
    return JSON.parse(data);
  } catch (error) {
    console.log(`Error ${error}`);
    return {};
  }
}

function saveProducts() {
    try {
        fs.writeFileSync("./products.json", JSON.stringify(products), "utf-8");
        console.log("Products saved successfully!")
    } catch (error) {
        console.log("Error", error)
    }
}


export const findAll = async () : Promise<UnitProduct[]> => Object.values(products)

export const findOne = async (id : string) : Promise<UnitProduct> => products[id]

export const create = async (productInfo : Product) : Promise<null | UnitProduct> => {

    let id = random()

    let product = await findOne(id)

    while (product) {
        id = random ()
        await findOne(id)
    }

    products[id] = {
        id : id,
        ...productInfo
    }

    saveProducts()

    return products[id]
}

export const update = async (id : string, updateValues : Product) : Promise<UnitProduct | null> => {

    const product = await findOne(id) 

    if (!product) {
        return null
    }

    products[id] = {
        id,
        ...updateValues
    }

    saveProducts()

    return products[id]
}

export const remove = async (id : string) : Promise<null | void> => {

    const product = await findOne(id)

    if (!product) {
        return null
    }

    delete products[id]

    saveProducts()

}

Again, you can reference the user's section for more details on what these functions provide to our API.

Once our logic checks out, it's time to implement the routes for our products.

Populate the ./src/products.routes.ts file with the following code :

import express, {Request, Response} from "express"
import { Product, UnitProduct } from "./product.interface"
import * as database from "./product.database"
import {StatusCodes} from "http-status-codes"

export const productRouter = express.Router()

productRouter.get('/products', async (req : Request, res : Response) => {
    try {
       const allProducts = await database.findAll()

       if (!allProducts) {
        return res.status(StatusCodes.NOT_FOUND).json({error : `No products found!`})
       }

       return res.status(StatusCodes.OK).json({total : allProducts.length, allProducts})
    } catch (error) {
       return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error}) 
    }
})

productRouter.get("/product/:id", async (req : Request, res : Response) => {
    try {
        const product = await database.findOne(req.params.id)

        if (!product) {
            return res.status(StatusCodes.NOT_FOUND).json({error : "Product does not exist"})
        }

        return res.status(StatusCodes.OK).json({product})
    } catch (error) {
        return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error})
    }
})


productRouter.post("/product", async (req : Request, res : Response) => {
    try {
        const {name, price, quantity, image} = req.body

        if (!name || !price || !quantity || !image) {
            return res.status(StatusCodes.BAD_REQUEST).json({error : `Please provide all the required parameters..`})
        }
        const newProduct = await database.create({...req.body})
        return res.status(StatusCodes.CREATED).json({newProduct})
    } catch (error) {
        return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error})
    }
})

productRouter.put("/product/:id", async (req : Request, res : Response) => {
    try {
        const id = req.params.id

        const newProduct = req.body

        const findProduct = await database.findOne(id)

        if (!findProduct) {
            return res.status(StatusCodes.NOT_FOUND).json({error : `Product does not exist..`})
        }

        const updateProduct = await database.update(id, newProduct)

        return res.status(StatusCodes.OK).json({updateProduct})
    } catch (error) {
        return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error})
    }
})


productRouter.delete("/product/:id", async (req : Request, res : Response) => {
    try {
        const getProduct = await database.findOne(req.params.id)

        if (!getProduct) {
            return res.status(StatusCodes.NOT_FOUND).json({error : `No product with ID ${req.params.id}`})
        }

        await database.remove(req.params.id)

        return res.status(StatusCodes.OK).json({msg : `Product deleted..`})

    } catch (error) {
        return res.status(StatusCodes.INTERNAL_SERVER_ERROR).json({error})
    }
})

Don't forget to import and call the product's route in our app.ts file, which should now look like this :

import express from "express"
import * as dotevnv from "dotenv"
import cors from "cors"
import helmet from "helmet"
import { userRouter } from "./users/users.routes"
import { productRouter } from "./products/product.routes"

dotevnv.config()

if (!process.env.PORT) {
    console.log(`No port value specified...`)
}

const PORT = parseInt(process.env.PORT as string, 10)

const app = express()

app.use(express.json())
app.use(express.urlencoded({extended : true}))
app.use(cors())
app.use(helmet())

app.use('/', userRouter)
app.use('/', productRouter)

app.listen(PORT, () => {
    console.log(`Server is listening on port ${PORT}`)
})

Perfect. We now have a full-fledged API built with Typescript and Nodejs. Hurray!!

Let's test our endpoints.

Create product

All products

Single product

Update product

Delete product

If you add new products they will be appended to the products.json file and it will look like this :

And that's we are done. If you came this far, Congratulations and Thank you!

Comments and recommendations are welcome.

You can find the complete code on github here -> GITHUB

BUILDING MY BLOGGING API (CHALLENGES FACED, LESSONS LEARNT AND OPTIMIZATION)

STEVE — Thu, 05 Jan 2023 11:42:11 +0000

The project at hand was a blogging API built with JavaScript, Node.js, Express, and MongoDB as part of a second semester exam. The initial scope of the project included various features such as creating and publishing blog posts, managing user accounts, and implementing authentication and authorization. However, during the development process, there were certain features that were either wrongly implemented or not implemented at all.

One of the main challenges faced while working on the project was ensuring the correctness and reliability of the implemented features. To address this, a decision was made to re-implement all the wrongly implemented features and to complete the implementation of all the features that were not implemented at all.

New Feature:

In addition to re-implementing the wrongly implemented features, a new feature was also added to the project – user validation using JOI. JOI (Javascript Object Schema validation) is a powerful and flexible schema validation library for JavaScript objects. It allows developers to define a schema for an object and then validate the object against that schema. This helps to ensure that the object adheres to the required format and structure, and also helps to prevent errors and bugs in the application.

The new feature was implemented by defining a schema for the user object using JOI, and then using the validate method of JOI to validate the user object against the schema. Any errors or deviations from the schema were then handled and appropriate feedback was provided to the user. This added an extra layer of security and reliability to the application, ensuring that only valid and properly formatted user data was processed and stored.

`const Joi = require('joi');

const userValidation = async (req, res, next) => {
try {
const payload = req.body
await schema.validateAsync(payload)
next()
} catch (error) {
console.log(error)
return res
.status(500)
.json({error : error.details[0].message})
}
}

const schema = Joi.object({
email: Joi.string()
.email({ minDomainSegments: 2, tlds: { allow: ['com', 'net'] }})
.required(),

password: Joi.string()
    .pattern(new RegExp('^[a-zA-Z0-9]{3,30}$'))

})

module.exports = userValidation`

Re-implementation of Wrongly Implemented Features:

One of the features that was wrongly implemented was the authentication and authorization system. The initial implementation had several security vulnerabilities and did not properly enforce the required permissions and access controls. To fix this, the authentication and authorization system was completely re-implemented from scratch.

The re-implementation process involved designing a new authentication and authorization flow that addressed the security vulnerabilities of the initial implementation. This included the use of secure hashes and salts for storing passwords, as well as the implementation of proper access controls and permissions. The re-implemented system was thoroughly tested to ensure that it was reliable and secure.

Another feature that was wrongly implemented was the user account management system. The initial implementation had several issues with data consistency and reliability, which led to errors and inconsistencies in the application. To fix this, the user account management system was re-implemented to ensure that all data was properly validated, stored, and retrieved. This included the implementation of proper data validation and error handling, as well as the use of transactions to ensure data consistency.

Implementation of Missing Features:

In addition to re-implementing the wrongly implemented features, there were also several features that were not implemented at all in the initial implementation. These features were completed as part of the re-implementation process.

One of the missing features that was implemented was the ability to delete blog posts. This feature was implemented by adding a new route and corresponding logic to the application, which allowed users with the appropriate permissions to delete their own blog posts. The delete functionality was thoroughly tested to ensure that it worked as expected.

Another missing feature that was implemented was the ability to edit blog posts. This feature was implemented by adding a new route and corresponding logic to the application, which allowed users with the appropriate permissions to edit their own blog posts. The edit functionality was also thoroughly tested to ensure that it worked.