DEV Community: Franklin Thaker

Avoiding console.log in Production: Best Practices for Robust Logging

Franklin Thaker — Sun, 08 Dec 2024 05:26:02 +0000

Introduction

Logging is crucial for debugging and monitoring applications, but improper logging can lead to performance issues, security vulnerabilities, and cluttered output. In this article, we'll explore why console.log should be avoided in production and provide best practices using examples.

Why one should avoid console.log in Production?

Performance Overhead -> This took around 46 seconds in my system.

console.time("with -> console.log");
for (let i = 0; i < 1000000; i++) {
    console.log(`Iteration number: ${i}`);
}
console.timeEnd("with -> console.log");

This loop logs a message a million times, causing performance degradation.

-> This took around 1ms in my system.

console.time("without -> console.log");
for (let i = 0; i < 1000000; i++) {
}
console.timeEnd("without -> console.log");

Security Risks Logging sensitive information can expose data to unintended parties. This code logs sensitive credentials, posing security risks.

const userCredentials = { username: 'john_doe', password: 's3cr3t' };
console.log(userCredentials);

Cluttered Logs Frequent logging can overwhelm the console, making it difficult to find relevant information.

function processOrder(order) {
  console.log('Processing order:', order);
  // Order processing logic here
  console.log('Order processed successfully');
}

Best Practices for Logging in Production

Use a Proper Logging Library Libraries like morgan, winston, pino, or log4js provide structured logging with log levels.

const pino = require('pino');
const logger = pino();

function processOrder(order) {
  logger.info({ order }, 'Processing order');
  // Order processing logic here
  logger.info('Order processed successfully');
}

Log Sensitive Information Securely Avoid logging sensitive data directly.

const userCredentials = { username: 'john_doe', password: 's3cr3t' };
logger.info({ username: userCredentials.username }, 'User logged in');

Implement Conditional Logging

const isProduction = process.env.NODE_ENV === 'production';

function log(message) {
  if (!isProduction) {
    console.log(message);
  }
}

log('This message will only appear in development');

Log to a Server or External Service

const axios = require('axios');

function logToServer(message) {
  axios.post('/api/log', { message })
    .catch(error => console.error('Failed to send log:', error));
}

logToServer('This is an important event');

Conclusion

Using console.log in production can lead to performance issues, security risks, and cluttered logs. By adopting proper logging practices with dedicated libraries and secure methodologies, you can ensure that your application is robust, maintainable, and secure.

Effortless Web Hosting with Caddy: A Beginner’s Guide

Franklin Thaker — Sat, 23 Nov 2024 05:35:02 +0000

In the ever-evolving world of web servers, Caddy has emerged as a game-changer. Designed for simplicity and modern workflows, it offers automatic HTTPS, straightforward configuration, and a developer-friendly approach.

What is Caddy Server?

Caddy is a modern, open-source web server written in Go. It’s known for its ability to handle complex web hosting tasks effortlessly. Whether you’re hosting a static site, reverse proxying APIs, or managing secure connections, Caddy simplifies the process with features like:

Automatic HTTPS: Get secure connections by default using Let's Encrypt.
Readable Configs: Manage sites with a simple Caddyfile—no cryptic syntax required.
Built-in Reverse Proxy: Handle API routing and load balancing without extra tools.

Why Choose Caddy?

Compared to traditional servers like Apache and Nginx, Caddy is:

Easier to Use: Minimal setup and automatic configurations.
Secure by Default: HTTPS is enabled automatically.
Lightweight Yet Powerful: Written in Go for speed and scalability.

Quick Setup

Download the version according to your OS -> https://caddyserver.com/download
Create a Caddyfile in your server's directory.

# Example Caddyfile

localhost {
    reverse_proxy localhost:8080
}

Start the Server:
- sudo caddy start -> to run the server in the background
- sudo caddy stop -> to stop the background running caddy server
- sudo caddy run -> to run the server in foreground

When to Use Caddy?

Static Websites: Host portfolios, blogs, or landing pages with ease.
APIs and Microservices: Use built-in reverse proxying for seamless API management.
HTTPS Needs: Say goodbye to manual SSL certificate management.

Quick Example:

// app.js
const express = require('express');
const app = express();
const port = 8080;

app.get('/', (req, res) => {
  res.send('Hello from Express server!');
});

app.listen(port, () => {
  console.log(`Server is running on http://localhost:${port}`);
});

# Caddyfile

localhost {
    reverse_proxy localhost:8080
}

node app.js -> This will start your actual backend server in 8080 PORT
sudo caddy run -> This will start the caddy server and you will be able to handle all requests at localhost:443

Extra tips

For Linux, once you get the binary from official site, you can rename that to "caddy" and then move the file to "/usr/local/bin/"
Then change the permissions. sudo chmod ugoa+x caddy
Then you will be able to access "caddy" via CLI anywhere in your system

Conclusion

Caddy server is perfect for developers looking for a modern, no-hassle web server. Whether you’re a beginner or a seasoned professional, its features and simplicity make it an excellent choice for most projects.

The Mind Games of Jigsaw: A list of His Most Famous Quotes

Franklin Thaker — Sat, 09 Nov 2024 06:49:09 +0000

Live or die. Make your choice.

The rules are simple.

I want to play a game.

Those who don’t appreciate life do not deserve life.

Cherish your life.

If you're good at anticipating the human mind, it leaves nothing to chance.

Heed my warning.

The choice is yours.

Experience is a harsh teacher.

You Can’t Help Them. They Have To Help Themselves.

This Is Not Retribution. This Is A Reawakening.

If You Find The Meaning Of Your Life, What You’ve Found Is Your Soul.

When Faced With Death, Who Should Live Versus Who Will Live Are Two Entirely Separate Things.

Congratulations, You Are Still Alive. Most People Are So Ungrateful To Be Alive... But Not You, Not Anymore.

Every Day Of Your Working Life You've Given People News That They'll Die. Now, You Will Be The Cause Of Death.

Death Is A Surprise Party. Unless Of Course, You're Already Dead On The Inside.

Game Over.

Mastering CRUD Operations with OpenSearch in Python: A Practical Guide

Franklin Thaker — Sat, 21 Sep 2024 12:48:22 +0000

OpenSearch, an open-source alternative to Elasticsearch, is a powerful search and analytics engine built to handle large datasets with ease. In this blog, we’ll demonstrate how to perform basic CRUD (Create, Read, Update, Delete) operations in OpenSearch using Python.

Prerequisites:

Python 3.7+
OpenSearch installed locally using Docker
Familiarity with RESTful APIs

Step 1: Setting Up OpenSearch Locally with Docker

To get started, we need a local OpenSearch instance. Below is a simple docker-compose.yml file that spins up OpenSearch and OpenSearch Dashboards.

version: '3'
services:
  opensearch-test-node-1:
    image: opensearchproject/opensearch:2.13.0
    container_name: opensearch-test-node-1
    environment:
      - cluster.name=opensearch-test-cluster
      - node.name=opensearch-test-node-1
      - discovery.seed_hosts=opensearch-test-node-1,opensearch-test-node-2
      - cluster.initial_cluster_manager_nodes=opensearch-test-node-1,opensearch-test-node-2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-test-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600
    networks:
      - opensearch-test-net

  opensearch-test-node-2:
    image: opensearchproject/opensearch:2.13.0
    container_name: opensearch-test-node-2
    environment:
      - cluster.name=opensearch-test-cluster
      - node.name=opensearch-test-node-2
      - discovery.seed_hosts=opensearch-test-node-1,opensearch-test-node-2
      - cluster.initial_cluster_manager_nodes=opensearch-test-node-1,opensearch-test-node-2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-test-data2:/usr/share/opensearch/data
    networks:
      - opensearch-test-net

  opensearch-test-dashboards:
    image: opensearchproject/opensearch-dashboards:2.13.0
    container_name: opensearch-test-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-test-node-1:9200","http://opensearch-test-node-2:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true"
    networks:
      - opensearch-test-net

volumes:
  opensearch-test-data1:
  opensearch-test-data2:

networks:
  opensearch-test-net:

Run the following command to bring up your OpenSearch instance:
docker-compose up
OpenSearch will be accessible at http://localhost:9200.

Step 2: Setting Up the Python Environment

python -m venv .venv
source .venv/bin/activate
pip install opensearch-py

We'll also structure our project as follows:

├── interfaces.py
├── main.py
├── searchservice.py
├── docker-compose.yml

Step 3: Defining Interfaces and Resources (interfaces.py)

In the interfaces.py file, we define our Resource and Resources classes. These will help us dynamically handle different resource types in OpenSearch (in this case, users).

from dataclasses import dataclass, field

@dataclass
class Resource:
    name: str

    def __post_init__(self) -> None:
        self.name = self.name.lower()

@dataclass
class Resources:
    users: Resource = field(default_factory=lambda: Resource("Users"))

Step 4: CRUD Operations with OpenSearch (searchservice.py)

In searchservice.py, we define an abstract class SearchService to outline the required operations. The HTTPOpenSearchService class then implements these CRUD methods, interacting with the OpenSearch client.

# coding: utf-8

import abc
import logging
import typing as t
from dataclasses import dataclass
from uuid import UUID

from interfaces import Resource, Resources
from opensearchpy import NotFoundError, OpenSearch

resources = Resources()


class SearchService(abc.ABC):
    def search(
        self,
        kinds: t.List[Resource],
        tenants_id: UUID,
        companies_id: UUID,
        query: t.Dict[str, t.Any],
    ) -> t.Dict[t.Literal["hits"], t.Dict[str, t.Any]]:
        raise NotImplementedError

    def delete_index(
        self,
        kind: Resource,
        tenants_id: UUID,
        companies_id: UUID,
        data: t.Dict[str, t.Any],
    ) -> None:
        raise NotImplementedError

    def index(
        self,
        kind: Resource,
        tenants_id: UUID,
        companies_id: UUID,
        data: t.Dict[str, t.Any],
    ) -> t.Dict[str, t.Any]:
        raise NotImplementedError

    def delete_document(
        self,
        kind: Resource,
        tenants_id: UUID,
        companies_id: UUID,
        document_id: str,
    ) -> t.Optional[t.Dict[str, t.Any]]:
        raise NotImplementedError

    def create_index(
        self,
        kind: Resource,
        tenants_id: UUID,
        companies_id: UUID,
        data: t.Dict[str, t.Any],
    ) -> None:
        raise NotImplementedError


@dataclass(frozen=True)
class HTTPOpenSearchService(SearchService):
    client: OpenSearch

    def _gen_index(
        self,
        kind: Resource,
        tenants_id: UUID,
        companies_id: UUID,
    ) -> str:
        return (
            f"tenant_{str(UUID(str(tenants_id)))}"
            f"_company_{str(UUID(str(companies_id)))}"
            f"_kind_{kind.name}"
        )

    def index(
        self,
        kind: Resource,
        tenants_id: UUID,
        companies_id: UUID,
        data: t.Dict[str, t.Any],
    ) -> t.Dict[str, t.Any]:
        self.client.index(
            index=self._gen_index(kind, tenants_id, companies_id),
            body=data,
            id=data.get("id"),
        )
        return data

    def delete_index(
        self,
        kind: Resource,
        tenants_id: UUID,
        companies_id: UUID,
    ) -> None:
        try:
            index = self._gen_index(kind, tenants_id, companies_id)
            if self.client.indices.exists(index):
                self.client.indices.delete(index)
        except NotFoundError:
            pass

    def create_index(
        self,
        kind: Resource,
        tenants_id: UUID,
        companies_id: UUID,
    ) -> None:
        body: t.Dict[str, t.Any] = {}
        self.client.indices.create(
            index=self._gen_index(kind, tenants_id, companies_id),
            body=body,
        )

    def search(
        self,
        kinds: t.List[Resource],
        tenants_id: UUID,
        companies_id: UUID,
        query: t.Dict[str, t.Any],
    ) -> t.Dict[t.Literal["hits"], t.Dict[str, t.Any]]:
        return self.client.search(
            index=",".join(
                [self._gen_index(kind, tenants_id, companies_id) for kind in kinds]
            ),
            body={"query": query},
        )

    def delete_document(
        self,
        kind: Resource,
        tenants_id: UUID,
        companies_id: UUID,
        document_id: str,
    ) -> t.Optional[t.Dict[str, t.Any]]:
        try:
            response = self.client.delete(
                index=self._gen_index(kind, tenants_id, companies_id),
                id=document_id,
            )
            return response
        except Exception as e:
            logging.error(f"Error deleting document: {e}")
            return None

Step 5: Implementing CRUD in Main (main.py)

In main.py, we demonstrate how to:

Create an index in OpenSearch.

Index documents with sample user data.

Search for documents based on a query.

Delete a document using its ID.

main.py

# coding=utf-8

import logging
import os
import typing as t
from uuid import uuid4

import searchservice
from interfaces import Resources
from opensearchpy import OpenSearch

resources = Resources()

logging.basicConfig(level=logging.INFO)

search_service = searchservice.HTTPOpenSearchService(
    client=OpenSearch(
        hosts=[
            {
                "host": os.getenv("OPENSEARCH_HOST", "localhost"),
                "port": os.getenv("OPENSEARCH_PORT", "9200"),
            }
        ],
        http_auth=(
            os.getenv("OPENSEARCH_USERNAME", ""),
            os.getenv("OPENSEARCH_PASSWORD", ""),
        ),
        use_ssl=False,
        verify_certs=False,
    ),
)

tenants_id: str = "f0835e2d-bd68-406c-99a7-ad63a51e9ef9"
companies_id: str = "bf58c749-c90a-41e2-b66f-6d98aae17a6c"
search_str: str = "frank"
document_id_to_delete: str = str(uuid4())

fake_data: t.List[t.Dict[str, t.Any]] = [
    {"id": document_id_to_delete, "name": "Franklin", "tech": "python,node,golang"},
    {"id": str(uuid4()), "name": "Jarvis", "tech": "AI"},
    {"id": str(uuid4()), "name": "Parry", "tech": "Golang"},
    {"id": str(uuid4()), "name": "Steve", "tech": "iOS"},
    {"id": str(uuid4()), "name": "Frank", "tech": "node"},
]

search_service.delete_index(
    kind=resources.users, tenants_id=tenants_id, companies_id=companies_id
)

search_service.create_index(
    kind=resources.users,
    tenants_id=tenants_id,
    companies_id=companies_id,
)

for item in fake_data:
    search_service.index(
        kind=resources.users,
        tenants_id=tenants_id,
        companies_id=companies_id,
        data=dict(tenants_id=tenants_id, companies_id=companies_id, **item),
    )

search_query: t.Dict[str, t.Any] = {
    "bool": {
        "must": [],
        "must_not": [],
        "should": [],
        "filter": [
            {"term": {"tenants_id.keyword": tenants_id}},
            {"term": {"companies_id.keyword": companies_id}},
        ],
    }
}
search_query["bool"]["must"].append(
    {
        "multi_match": {
            "query": search_str,
            "type": "phrase_prefix",
            "fields": ["name", "tech"],
        }
    }
)

search_results = search_service.search(
    kinds=[resources.users],
    tenants_id=tenants_id,
    companies_id=companies_id,
    query=search_query,
)

final_result = search_results.get("hits", {}).get("hits", [])
for item in final_result:
    logging.info(["Item -> ", item.get("_source", {})])

deleted_result = search_service.delete_document(
    kind=resources.users,
    tenants_id=tenants_id,
    companies_id=companies_id,
    document_id=document_id_to_delete,
)
logging.info(["Deleted result -> ", deleted_result])

Step 6: Running the project

docker compose up
python main.py

Results:

It should print found & deleted records information.

Step 7: Conclusion

In this blog, we’ve demonstrated how to set up OpenSearch locally using Docker and perform basic CRUD operations with Python. OpenSearch provides a powerful and scalable solution for managing and querying large datasets. While this guide focuses on integrating OpenSearch with dummy data, in real-world applications, OpenSearch is often used as a read-optimized store for faster data retrieval. In such cases, it is common to implement different indexing strategies to ensure data consistency by updating both the primary database and OpenSearch concurrently.

This ensures that OpenSearch remains in sync with your primary data source, optimizing both performance and accuracy in data retrieval.

References:

https://github.com/FranklinThaker/opensearch-integration-example

Unleashing MongoDB: Why Cursor-Based Pagination Outperforms Offset-Based Pagination Every Time!

Franklin Thaker — Wed, 04 Sep 2024 13:20:48 +0000

Pagination is a critical part of any database operation when dealing with large datasets. It allows you to split data into manageable chunks, making it easier to browse, process, and display. MongoDB provides two common pagination methods: offset-based and cursor-based. While both methods serve the same purpose, they differ significantly in performance and usability, especially as the dataset grows.

Let's dive into the two approaches and see why cursor-based pagination often outperforms offset-based pagination.

1. Offset-Based Pagination

Offset-based pagination is straightforward. It retrieves a specific number of records starting from a given offset. For example, the first page might retrieve records 0-9, the second page retrieves records 10-19, and so on.

However, this method has a significant drawback: as you move to higher pages, the query becomes slower. This is because the database needs to skip over the records from the previous pages, which involves scanning through them.

Here’s the code for offset-based pagination:

async function offset_based_pagination(params) {
  const { page = 5, limit = 100 } = params;
  const skip = (page - 1) * limit;
  const results = await collection.find({}).skip(skip).limit(limit).toArray();
  console.log(`Offset-based pagination (Page ${page}):`, results.length, "page", page, "skip", skip, "limit", limit);
}

2. Cursor-Based Pagination

Cursor-based pagination, also known as keyset pagination, relies on a unique identifier (e.g., an ID or timestamp) to paginate through the records. Instead of skipping a certain number of records, it uses the last retrieved record as the reference point for fetching the next set.

This approach is more efficient because it avoids the need to scan the records before the current page. As a result, the query time remains consistent, regardless of how deep into the dataset you go.

Here's the code for cursor-based pagination:

async function cursor_based_pagination(params) {
  const { lastDocumentId, limit = 100 } = params;
  const query = lastDocumentId ? { documentId: { $gt: lastDocumentId } } : {};
  const results = await collection
    .find(query)
    .sort({ documentId: 1 })
    .limit(limit)
    .toArray();
  console.log("Cursor-based pagination:", results.length);
}

In this example, lastDocumentId is the ID of the last document from the previous page. When querying for the next page, the database fetches documents with an ID greater than this value, ensuring a seamless transition to the next set of records.

3. Performance Comparison

Let’s see how these two methods perform with a large dataset.

async function testMongoDB() {
    console.time("MongoDB Insert Time:");
    await insertMongoDBRecords();
    console.timeEnd("MongoDB Insert Time:");

  // Create an index on the documentId field
  await collection.createIndex({ documentId: 1 });
  console.log("Index created on documentId field");

  console.time("Offset-based pagination Time:");
  await offset_based_pagination({ page: 2, limit: 250000 });
  console.timeEnd("Offset-based pagination Time:");

  console.time("Cursor-based pagination Time:");
  await cursor_based_pagination({ lastDocumentId: 170000, limit: 250000 });
  console.timeEnd("Cursor-based pagination Time:");

  await client.close();
}

In the performance test, you’ll notice that the offset-based pagination takes longer as the page number increases, whereas the cursor-based pagination remains consistent, making it the better choice for large datasets. This example also demonstrates the power of indexing as well. Try to remove index & then see the result as well!

Why Indexing is Important

Without an index, MongoDB would need to perform a collection scan, which means it has to look at each document in the collection to find the relevant data. This is inefficient, especially when your dataset grows. Indexes allow MongoDB to efficiently to find the documents that match your query conditions, significantly speeding up query performance.

In the context of cursor-based pagination, the index ensures that fetching the next set of documents (based on documentId) is quick and does not degrade in performance as more documents are added to the collection.

Conclusion

While offset-based pagination is easy to implement, it can become inefficient with large datasets due to the need to scan through records. Cursor-based pagination, on the other hand, provides a more scalable solution, keeping performance consistent regardless of the dataset size. If you are working with large collections in MongoDB, it’s worth considering cursor-based pagination for a smoother and faster experience.

Here's complete index.js for you to run locally:

const { MongoClient } = require("mongodb");
const uri = "mongodb://localhost:27017";
const client = new MongoClient(uri);
client.connect();
const db = client.db("testdb");
const collection = db.collection("testCollection");

async function insertMongoDBRecords() {
  try {
    let bulkOps = [];

    for (let i = 0; i < 2000000; i++) {
      bulkOps.push({
        insertOne: {
          documentId: i,
          name: `Record-${i}`,
          value: Math.random() * 1000,
        },
      });

      // Execute every 10000 operations and reinitialize
      if (bulkOps.length === 10000) {
        await collection.bulkWrite(bulkOps);
        bulkOps = [];
      }
    }

    if (bulkOps.length > 0) {
      await collection.bulkWrite(bulkOps);
      console.log("🚀 Inserted records till now -> ", bulkOps.length);
    }

    console.log("MongoDB Insertion Completed");
  } catch (err) {
    console.error("Error in inserting records", err);
  }
}

async function offset_based_pagination(params) {
  const { page = 5, limit = 100 } = params;
  const skip = (page - 1) * limit;
  const results = await collection.find({}).skip(skip).limit(limit).toArray();
  console.log(`Offset-based pagination (Page ${page}):`, results.length, "page", page, "skip", skip, "limit", limit);
}

async function cursor_based_pagination(params) {
  const { lastDocumentId, limit = 100 } = params;
  const query = lastDocumentId ? { documentId: { $gt: lastDocumentId } } : {};
  const results = await collection
    .find(query)
    .sort({ documentId: 1 })
    .limit(limit)
    .toArray();
  console.log("Cursor-based pagination:", results.length);
}

async function testMongoDB() {
  console.time("MongoDB Insert Time:");
  await insertMongoDBRecords();
  console.timeEnd("MongoDB Insert Time:");

  // Create an index on the documentId field
  await collection.createIndex({ documentId: 1 });
  console.log("Index created on documentId field");

  console.time("Offset-based pagination Time:");
  await offset_based_pagination({ page: 2, limit: 250000 });
  console.timeEnd("Offset-based pagination Time:");

  console.time("Cursor-based pagination Time:");
  await cursor_based_pagination({ lastDocumentId: 170000, limit: 250000 });
  console.timeEnd("Cursor-based pagination Time:");

  await client.close();
}

testMongoDB();

Real-time Data Updates with Server-Sent Events (SSE) in Node.js

Franklin Thaker — Fri, 05 Jul 2024 06:15:46 +0000

In this blog post, we'll explore how to use Server-Sent Events (SSE) to push real-time data from a server to clients. We'll create a simple example using Node.js and Express to demonstrate how SSE works.

What are Server-Sent Events (SSE)?

Server-Sent Events (SSE) allow servers to push updates to the client over a single, long-lived HTTP connection. Unlike WebSockets, SSE is a unidirectional protocol where updates flow from server to client. This makes SSE ideal for live data feeds like news updates, stock prices, or notifications.

Creating the Server

// app.js
const express = require("express");
const app = express();
const { v4 } = require("uuid");
let clients = [];

app.use(express.json());
app.use(express.static("./public"));

function sendDataToAllClients() {
  const value_to_send_to_all_clients = Math.floor(Math.random() * 1000) + 1;
  clients.forEach((client) =>
    client.response.write("data: " + value_to_send_to_all_clients + "\n\n")
  );
}

app.get("/subscribe", async (req, res) => {
  const clients_id = v4();
  const headers = {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    Connection: "keep-alive",
  };
  res.writeHead(200, headers);
  clients.push({ id: clients_id, response: res });

  // Close the connection when the client disconnects
  req.on("close", () => {
    clients = clients.filter((c) => c.id !== clients_id);
    console.log(`${clients_id} Connection closed`);
    res.end("OK");
  });
});

app.get("/data", (req, res) => {
  sendDataToAllClients();
  res.send("Data sent to all subscribed clients.");
});

app.listen(80, () => {
  console.log("Server is running on port 80");
});

Code Explanation

Express Setup: We create an Express app and set up JSON parsing and static file serving.
Client Management: We maintain a list of connected clients.
SSE Headers: In the /subscribe endpoint, we set the necessary headers to establish an SSE connection.
Send Data: The sendDataToAllClients function sends random data to all subscribed clients.
Subscribe Endpoint: Clients connect to this endpoint to receive real-time updates.
Data Endpoint: This endpoint triggers the sendDataToAllClients function to send data.

Creating the Client

Next, let's create a simple HTML page to subscribe to the server and display the real-time data.

<!-- index.html  -->
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>SSE - Example (Server-Sent-Events)</title>
</head>
<body>
  <div id="data"></div>
</body>
</html>

<script>
  const subscription = new EventSource("/subscribe");

  // Default events
  subscription.addEventListener("open", () => {
    console.log("Connection opened");
  });

  subscription.addEventListener("error", () => {
    console.error("Subscription err'd");
    subscription.close();
  });

  subscription.addEventListener("message", (event) => {
    console.log("Receive message", event);
    document.getElementById("data").innerHTML += `${event.data}<br>`;
  });
</script>

Code Explanation

EventSource: We create a new EventSource object to subscribe to the /subscribe endpoint.
Event Listeners: We set up listeners for open, error, and message events.
Display Data: When a message is received, we append the data to a div.

Running the Example

Start the server:
node app.js
Open your browser and navigate to http://localhost/subscribe. [Don't close it]
Now, open another tab & navigate to http://localhost/data and you should see random data prints onto the screen in other tab.
You can subscribe to multiple clients/tabs as you want & you can simply navigate to http://localhost/data & you would see the same data emits to all the subscribed clients.

Conclusion

In this post, we've seen how to use Server-Sent Events (SSE) to push real-time updates from a server to connected clients using Node.js and Express. SSE is a simple yet powerful way to add real-time capabilities to your web applications without the complexity of WebSockets.

Warning⚠️:

When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6). The issue has been marked as "Won't fix" in Chrome and Firefox. This limit is per browser + domain, which means that you can open 6 SSE connections across all of the tabs to www.example1.com and another 6 SSE connections to www.example2.com (per Stackoverflow). When using HTTP/2, the maximum number of simultaneous HTTP streams is negotiated between the server and the client (defaults to 100).

Happy coding!

Resources:

https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

How to use SRI (Subresource integrtiy) attribute in script tag to prevent modification of static resources ?

Franklin Thaker — Sat, 29 Jun 2024 06:23:44 +0000

Understanding Supply Chain Attacks

Supply chain attacks involve compromising a third-party service or library to inject malicious code into websites that rely on it.

How Could This Be Prevented?

One effective way to mitigate such risks is by using the Subresource Integrity (SRI) attribute in your HTML scripts and link tags. SRI allows browsers to verify that the fetched resources (like scripts or stylesheets) are delivered without unexpected manipulation.

Demonstration

#script.js
console.log("Hello World - Secured!");

#index.html
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <script
    src="script.js"
    integrity="sha384-[YOUR-GENERATED-HASH]"
    crossorigin="anonymous"
  ></script>
  <title>SUBRESOURCE INTEGRITY EXAMPLE</title>
</head>
<body>
  Hello World
</body>
</html>

#app.js - to serve above files
const express = require('express');
const app = express();

app.use(express.static("./"));

app.listen(80, () => {
  console.log('Server is running on http://localhost');
});

Generating the Integrity Hash

openssl dgst -sha384 -binary script.js | openssl base64 -A

Now if we perform all the above steps correctly then the following working output should appear:

Now Let's try to modify script.js

#script.js - modified
console.log("Hello World - Modified!");

Now try to open http://localhost/index.html, you should see following error:

None of the sha384 hashes in the integrity attribute match the content of the resource.

SO THE MODIFIEFD SCRIPT WON'T RUN AT ALL!!

How to disable camera devices in Linux systems ?

Franklin Thaker — Fri, 07 Jun 2024 13:47:03 +0000

lsusb
lsmod [to identify camera devices]

sudo nano /etc/modprobe.d/blacklist.conf

****add the following lines in above file****
##Disable webcam.
blacklist uvcvideo

References: https://forums.linuxmint.com/viewtopic.php?t=370960

Auth0 integration - Node.js + ExpressJS

Franklin Thaker — Tue, 04 Jun 2024 13:24:18 +0000

This is a simple guide to demonstrate backend Auth0 integration. There will be no frontend involved. User sign-up, log-in, log-out, all operations will be done through backend only.

// index.js
require('dotenv').config();
const { auth, requiresAuth } = require("express-openid-connect");
const app = require("express")();

const config = {
  authRequired: false,
  auth0Logout: true,
  secret: process.env.CLIENT_SECRET,
  baseURL: "http://localhost:3000",
  clientID: process.env.CLIENT_ID,
  issuerBaseURL:`https://${process.env.AUTH0_TENANT}.auth0.com`,
};

// auth router attaches /login, /logout, and /callback routes to the baseURL
app.use(auth(config));

// req.isAuthenticated is provided from the auth router
app.get("/", (req, res) => {
  res.send(req.oidc.isAuthenticated() ? "Logged in" : "Logged out");
});

app.get("/profile", requiresAuth(), (req, res) => {
  res.send(JSON.stringify(req.oidc.user));
});

app.listen(3000);

Environment Variables

To run this project, you will need to add the following environment variables to your .env file

CLIENT_ID -> Go to Auth0 -> Applications -> Settings -> Client ID

AUTH0_TENANT -> Go to Auth0 -> Applications -> Settings -> Domain

CLIENT_SECRET -> Run this command to generate the secret value:

openssl rand -hex 32

If you are running on Windows: Try to run this in Git Bash it should work without you needing to install Win64 OpenSSL
Also make sure to setup this in Settings tab in Auth0:

Allowed Callback URLs: http://localhost:3000
Allowed Logout URLs: http://localhost:3000

References
https://github.com/FranklinThaker/auth0-integration-nodejs
https://auth0.github.io/express-openid-connect/index.html

How to use compression in Node.js server for better bandwidth ?

Franklin Thaker — Wed, 15 May 2024 11:57:14 +0000

const express = require("express");
const compression = require("compression");
const app = express();

app.set("etag", false);
app.use(compression());

app.get("/data", (req, res) => {
  return res.json({
    message: "Hello, Axel Blaze, This is a test message. ".repeat(10000),
  });
});

app.listen(3000, function () {
  console.log("listening on 3000");
});

How to start your server & check if gzip compression is working or not!

DEBUG=compression node app.js

Tip

Make sure you pass correct Request header i.e. Accept-Encoding: gzip

Output examples:

With Compression

Without Compression

Uninterrupted Access to Dropbox API

Franklin Thaker — Sun, 12 May 2024 05:12:36 +0000

This guide will help you set up uninterrupted access to the Dropbox API using OAuth 2.0 and refresh tokens.

1. Authorize Your App

Visit the following URL to authorize your app and generate an authorization code:

https://www.dropbox.com/oauth2/authorize?client_id=YOUR_APP_KEY&response_type=code&token_access_type=offline

2. Use the Authorization Code

Once you've authorized your app, you'll receive an authorization code. Use this code in your app codebase, replacing ACCESS_CODE_FROM_STEP_1.

3. Start the Application

Run your application (e.g., app.js). For the first time, the application will generate a refresh token.

4. Store the Refresh Token

The refresh token generated in Step 3 won't expire. Make sure to store this long-term token securely. In this example, we're storing it as REFRESH_TOKEN.

5. Restart the Application

Restart your application. You can now use the upload functionality without any interruption.

Ref: https://github.com/FranklinThaker/Dropbox-API-Uninterrupted-Access