Akhilesh

Posted on Feb 13

The Ultimate Node.js Backend Mastery Guide: Zero to Production Hero

#backend #mern #node #webdev

Master Every Backend Concept - From Basics to Advanced Production Patterns

Note: This comprehensive guide covers everything you need to know about Node.js backend development. Whether you're preparing for interviews or building production applications, this single resource has you covered.

📑 Table of Contents

Introduction to Node.js
JavaScript ES6+ Essentials
Node.js Architecture & Event Loop
Core Modules Complete Guide
NPM & Package Management
Express.js Framework
REST API Design
Database Integration
Authentication & Authorization
Session Management
File Upload & Storage
Error Handling
Input Validation
Security Best Practices
Rate Limiting & Throttling
Caching Strategies
WebSockets & Real-Time
Message Queues
Email & Notifications
Payment Integration
Logging & Monitoring
Testing Strategies
Microservices
GraphQL APIs
Docker & Containers
CI/CD Pipelines
Deployment
Performance Optimization

1. Introduction to Node.js

What is Node.js?

Definition: Node.js is a JavaScript runtime built on Chrome's V8 engine that executes JavaScript code outside the browser. It enables server-side programming using JavaScript.

Key Features:

Event-Driven: Uses events to handle operations
Non-Blocking I/O: Doesn't wait for operations to complete
Single-Threaded: Uses one thread with event loop
Fast: V8 engine compiles JS to machine code
Cross-Platform: Runs on Windows, Mac, Linux

Why Node.js?

Traditional blocking code requires sequential execution. If Request 1 takes 5 seconds, Request 2 must wait, resulting in 10 seconds total for 2 requests.
Node.js uses non-blocking code. Request 1 starts and delegates to the OS, allowing Request 2 to start immediately. Both complete in approximately 5 seconds in parallel

// Traditional blocking code (other languages)
// Request 1 starts -> waits 5 seconds -> completes
// Request 2 waits for Request 1 to finish
// Total time: 10 seconds for 2 requests

// Node.js non-blocking code
// Request 1 starts -> delegates to OS -> continues
// Request 2 starts -> delegates to OS -> continues
// Both complete in ~5 seconds (parallel)

const http = require('http');

http.createServer((req, res) => {
    // This handles thousands of concurrent requests
    res.end('Hello World');
}).listen(3000);

Architecture Diagram

┌─────────────────────────────────────────┐
│         JavaScript Application          │
└─────────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────────┐
│          Node.js Runtime                │
│  ┌───────────────────────────────┐      │
│  │     V8 JavaScript Engine      │      │
│  │  (Google Chrome's JS Engine)  │      │
│  └───────────────────────────────┘      │
│  ┌───────────────────────────────┐      │
│  │        Event Loop             │      │
│  │   (Single-threaded magic)     │      │
│  └───────────────────────────────┘      │
│  ┌───────────────────────────────┐      │
│  │      Thread Pool (libuv)      │      │
│  │    (4 threads by default)     │      │
│  └───────────────────────────────┘      │
│  ┌───────────────────────────────┐      │
│  │       C++ Bindings            │      │
│  │    (Low-level operations)     │      │
│  └───────────────────────────────┘      │
└─────────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────────┐
│        Operating System                 │
│   (File System, Network, etc.)          │
└─────────────────────────────────────────┘

Installation

# Check if Node.js is installed
node --version
npm --version

# Install Node.js (visit nodejs.org)
# Download LTS version

# Using nvm (Node Version Manager) - Recommended
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
nvm install node
nvm use node

# First Node.js program
# Create app.js
console.log('Hello from Node.js!');

# Run it
node app.js

Global Objects

Node.js provides several global objects that are available in all modules.

// Current directory and filename
console.log(__dirname);  // /Users/username/project
console.log(__filename); // /Users/username/project/app.js

// Process information
console.log(process.env); // Environment variables
console.log(process.argv); // Command line arguments
console.log(process.pid); // Process ID
console.log(process.version); // Node version
console.log(process.platform); // Operating system

// Module exports
// math.js
module.exports = {
    add: (a, b) => a + b,
    subtract: (a, b) => a - b
};

// app.js
const math = require('./math');
console.log(math.add(5, 3)); // 8

2. JavaScript ES6+ Essentials

Variable Declarations

var - Function Scoped (Avoid in modern code)

var oldWay = 'I am function scoped';

function test() {
    var x = 10;
    if (true) {
        var x = 20; // Same variable
        console.log(x); // 20
    }
    console.log(x); // 20
}

let - Block Scoped

let count = 0;
if (true) {
    let count = 10; // Different variable
    console.log(count); // 10
}
console.log(count); // 0

const - Block Scoped, Immutable Binding

const API_KEY = 'secret123';
// API_KEY = 'new'; // Error

const user = { name: 'John' };
user.name = 'Jane'; // OK (object is mutable)
user.age = 30; // OK
// user = {}; // Error (binding is immutable)

const arr = [1, 2, 3];
arr.push(4); // OK
// arr = []; // Error

** Difference between let, const, and var?**

Feature	var	let	const
Scope	Function	Block	Block
Hoisting	Yes (undefined)	Yes (TDZ)	Yes (TDZ)
Re-declaration	✓	✗	✗
Re-assignment	✓	✓	✗
Best Use	Legacy code	Changing values	Default choice

Arrow Functions

Traditional Function vs Arrow Function

// Traditional function
function add(a, b) {
    return a + b;
}

// Arrow function
const add = (a, b) => a + b;

// Multiple statements
const multiply = (a, b) => {
    const result = a * b;
    console.log(`Result: ${result}`);
    return result;
};

// Single parameter (parentheses optional)
const square = x => x * x;

// No parameters
const greet = () => console.log('Hello');

// Returning object (wrap in parentheses)
const createUser = (name, age) => ({ name, age });

IMPORTANT: Arrow functions don't bind 'this'

const timer = {
    seconds: 0,
    start: function() {
        setInterval(() => {
            this.seconds++; // 'this' refers to timer
            console.log(this.seconds);
        }, 1000);
    }
};

// Regular function loses context
const timer2 = {
    seconds: 0,
    start: function() {
        setInterval(function() {
            this.seconds++; // 'this' is undefined
        }, 1000);
    }
};

When NOT to use arrow functions:

Object methods (need this)
Constructors (can't use new)
When you need 'arguments' object

Template Literals

const name = 'John';
const age = 30;

// Old way
const msg1 = 'Hello, ' + name + '! You are ' + age + ' years old.';

// Template literal
const msg2 = `Hello, ${name}! You are ${age} years old.`;

// Multi-line strings
const html = `
    <div>
        <h1>${name}</h1>
        <p>Age: ${age}</p>
        <p>Birth Year: ${new Date().getFullYear() - age}</p>
    </div>
`;

// Expression evaluation
const price = 19.99;
const tax = 0.08;
const total = `Total: $${(price * (1 + tax)).toFixed(2)}`;
console.log(total); // Total: $21.59

Tagged templates (advanced)

function sql(strings, ...values) {
    return strings.reduce((query, str, i) => {
        const value = values[i];
        const escaped = typeof value === 'string' 
            ? `'${value.replace(/'/g, "''")}'` 
            : value;
        return query + str + (escaped || '');
    }, '');
}

const userId = "1' OR '1'='1";
const query = sql`SELECT * FROM users WHERE id = ${userId}`;
// Prevents SQL injection

Destructuring

Array Destructuring

const colors = ['red', 'green', 'blue', 'yellow'];

const [first, second] = colors; // first='red', second='green'
const [, , third] = colors; // third='blue'
const [primary, ...rest] = colors; // rest=['green','blue','yellow']

// Default values
const [a = 1, b = 2, c = 3] = [10]; // a=10, b=2, c=3

// Swapping variables
let x = 1, y = 2;
[x, y] = [y, x]; // x=2, y=1

Object destructuring

const user = {
    id: 1,
    name: 'John Doe',
    email: 'john@example.com',
    age: 30,
    address: {
        city: 'New York',
        country: 'USA'
    }
};

const { name, email } = user;
const { age: userAge } = user; // Rename
const { phone = 'N/A' } = user; // Default
const { address: { city } } = user; // Nested

// Function parameters
function displayUser({ name, email, age = 0 }) {
    console.log(`${name} (${age}): ${email}`);
}
displayUser(user);

// Rest in objects
const { id, ...userWithoutId } = user;
console.log(userWithoutId); // Everything except id

Spread & Rest Operators

Spread Operator (...) - Expands Elements

// Arrays
const arr1 = [1, 2, 3];
const arr2 = [4, 5, 6];

const combined = [...arr1, ...arr2]; // [1,2,3,4,5,6]
const withExtra = [0, ...arr1, 3.5, ...arr2, 7]; // [0,1,2,3,3.5,4,5,6,7]

// Copy array (shallow)
const copy = [...arr1]; // Creates new array

// Objects
const user = { name: 'John', age: 30 };
const address = { city: 'NYC', country: 'USA' };

const fullUser = { ...user, ...address };
// { name: 'John', age: 30, city: 'NYC', country: 'USA' }

// Override properties
const updated = { ...user, age: 31 }; // { name: 'John', age: 31 }

Rest Operator (...) - Collects Elements

function sum(...numbers) {
    return numbers.reduce((total, num) => total + num, 0);
}

console.log(sum(1, 2, 3)); // 6
console.log(sum(1, 2, 3, 4, 5)); // 15

// With other parameters
function greet(greeting, ...names) {
    return `${greeting} ${names.join(', ')}!`;
}

console.log(greet('Hello', 'John', 'Jane', 'Bob'));
// Hello John, Jane, Bob!

Promises

Creating a Promise

const fetchData = () => {
    return new Promise((resolve, reject) => {
        setTimeout(() => {
            const success = true;
            if (success) {
                resolve({ data: 'User data' });
            } else {
                reject(new Error('Failed to fetch'));
            }
        }, 1000);
    });
};

// Using Promise
fetchData()
    .then(result => {
        console.log(result.data);
        return result.data;
    })
    .then(data => {
        console.log('Processing:', data);
    })
    .catch(error => {
        console.error('Error:', error.message);
    })
    .finally(() => {
        console.log('Cleanup');
    });

Promise Methods

const promise1 = Promise.resolve(3);
const promise2 = new Promise(resolve => setTimeout(() => resolve(42), 100));
const promise3 = fetch('https://api.example.com/data');

// Promise.all - Wait for all promises
Promise.all([promise1, promise2, promise3])
    .then(([result1, result2, result3]) => {
        console.log(result1, result2, result3);
    })
    .catch(error => {
        console.error('One failed:', error);
    });

// Promise.race - First to complete
Promise.race([promise1, promise2])
    .then(result => {
        console.log('Winner:', result);
    });

// Promise.allSettled - Wait for all, get all results
Promise.allSettled([promise1, promise2, promise3])
    .then(results => {
        results.forEach(result => {
            if (result.status === 'fulfilled') {
                console.log('Success:', result.value);
            } else {
                console.log('Failed:', result.reason);
            }
        });
    });

// Promise.any - First successful
Promise.any([promise1, promise2, promise3])
    .then(result => {
        console.log('First success:', result);
    })
    .catch(error => {
        console.error('All failed');
    });

Async/Await

Converting Promise to Async/Await

async function getData() {
    try {
        const result = await fetchData();
        console.log(result.data);
        return result;
    } catch (error) {
        console.error('Error:', error.message);
        throw error;
    } finally {
        console.log('Cleanup');
    }
}

// Multiple awaits (sequential)
async function processUsers() {
    const user = await fetchUser(1);
    const posts = await fetchUserPosts(user.id);
    const comments = await fetchPostComments(posts[0].id);
    return { user, posts, comments };
}

// Parallel execution
async function processUsersParallel() {
    const [user, posts, stats] = await Promise.all([
        fetchUser(1),
        fetchPosts(),
        fetchStats()
    ]);
    return { user, posts, stats };
}

// Error handling patterns
async function robustFetch() {
    try {
        const response = await fetch('https://api.example.com');
        if (!response.ok) {
            throw new Error(`HTTP error! status: ${response.status}`);
        }
        const data = await response.json();
        return data;
    } catch (error) {
        console.error('Fetch failed:', error);
        return null; // or throw
    }
}

// Async IIFE (Immediately Invoked Function Expression)
(async () => {
    const data = await getData();
    console.log(data);
})();

// Top-level await (Node.js 14.8+, ES modules)
// main.mjs
const data = await fetchData();
console.log(data);

Array Methods

const numbers = [1, 2, 3, 4, 5];

// map - Transform each element
const doubled = numbers.map(n => n * 2);
// [2, 4, 6, 8, 10]

// filter - Keep elements that pass test
const evens = numbers.filter(n => n % 2 === 0);
// [2, 4]

// reduce - Reduce to single value
const sum = numbers.reduce((acc, n) => acc + n, 0);
// 15

const max = numbers.reduce((max, n) => n > max ? n : max);
// 5

// find - First element that matches
const found = numbers.find(n => n > 3);
// 4

// findIndex - Index of first match
const index = numbers.findIndex(n => n > 3);
// 3

// some - Check if any element matches
const hasEven = numbers.some(n => n % 2 === 0);
// true

// every - Check if all elements match
const allPositive = numbers.every(n => n > 0);
// true

// forEach - Execute function for each
numbers.forEach((n, index) => {
    console.log(`Index ${index}: ${n}`);
});

// includes - Check if value exists
const hasThree = numbers.includes(3);
// true

// flat - Flatten nested arrays
const nested = [1, [2, 3], [4, [5, 6]]];
const flat = nested.flat(); // [1, 2, 3, 4, [5, 6]]
const deepFlat = nested.flat(2); // [1, 2, 3, 4, 5, 6]

// flatMap - map + flat
const words = ['hello world', 'foo bar'];
const letters = words.flatMap(word => word.split(' '));
// ['hello', 'world', 'foo', 'bar']

// Chaining methods
const result = numbers
    .filter(n => n > 2)
    .map(n => n * 2)
    .reduce((sum, n) => sum + n, 0);
// 24 (3+4+5 filtered, doubled to 6+8+10)

Object Methods

const user = {
    name: 'John',
    age: 30,
    email: 'john@example.com'
};

// Object.keys - Get all keys
const keys = Object.keys(user);
// ['name', 'age', 'email']

// Object.values - Get all values
const values = Object.values(user);
// ['John', 30, 'john@example.com']

// Object.entries - Get key-value pairs
const entries = Object.entries(user);
// [['name', 'John'], ['age', 30], ['email', 'john@example.com']]

// Object.fromEntries - Create object from entries
const newUser = Object.fromEntries([
    ['name', 'Jane'],
    ['age', 25]
]);
// { name: 'Jane', age: 25 }

// Object.assign - Merge objects
const defaults = { theme: 'dark', lang: 'en' };
const settings = Object.assign({}, defaults, user);
// { theme: 'dark', lang: 'en', name: 'John', age: 30, email: '...' }

// Better: Use spread
const settings2 = { ...defaults, ...user };

// Object.freeze - Make immutable
const frozen = Object.freeze({ value: 42 });
frozen.value = 100; // Silently fails (throws in strict mode)

// Object.seal - Prevent adding/removing properties
const sealed = Object.seal({ value: 42 });
sealed.value = 100; // OK
sealed.newProp = 'test'; // Fails
delete sealed.value; // Fails

// Optional chaining
const userData = {
    user: {
        address: {
            city: 'NYC'
        }
    }
};

const city = userData?.user?.address?.city; // 'NYC'
const zip = userData?.user?.address?.zip; // undefined (no error)

// Nullish coalescing
const value1 = null ?? 'default'; // 'default'
const value2 = 0 ?? 'default'; // 0 (not null/undefined)
const value3 = '' ?? 'default'; // '' (not null/undefined)

3. Node.js Architecture & Event Loop

Event Loop

Definition: The Event Loop is what allows Node.js to perform non-blocking I/O operations despite JavaScript being single-threaded. It continuously monitors the call stack and callback queues, executing callbacks when the stack is empty.

Event Loop Phases

   ┌───────────────────────────┐
┌─>│           timers          │  setTimeout/setInterval callbacks
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
│  │     pending callbacks     │  I/O callbacks deferred to next loop
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
│  │       idle, prepare       │  Internal use only
│  └─────────────┬─────────────┘      ┌───────────────┐
│  ┌─────────────┴─────────────┐      │   incoming:   │
│  │           poll            │<─────┤  connections, │
│  └─────────────┬─────────────┘      │   data, etc.  │
│  ┌─────────────┴─────────────┐      └───────────────┘
│  │           check           │  setImmediate() callbacks
│  └─────────────┬─────────────┘
│  ┌─────────────┴─────────────┐
└──┤      close callbacks      │  socket.on('close', ...)
   └───────────────────────────┘

Event Loop Example

console.log('1: Start');

setTimeout(() => {
    console.log('2: setTimeout 0ms');
}, 0);

setImmediate(() => {
    console.log('3: setImmediate');
});

Promise.resolve().then(() => {
    console.log('4: Promise');
});

process.nextTick(() => {
    console.log('5: nextTick');
});

console.log('6: End');

// Output order:
// 1: Start
// 6: End
// 5: nextTick (nextTick queue - highest priority)
// 4: Promise (microtask queue)
// 2: setTimeout 0ms (timer queue)
// 3: setImmediate (check queue)

Execution Order:

Call Stack - Synchronous code
process.nextTick queue
Microtask queue (Promises)
Timer queue (setTimeout/setInterval)
I/O callbacks
setImmediate queue
Close callbacks

Understanding Non-Blocking I/O

Blocking (Synchronous) - Bad for servers

const fs = require('fs');

console.log('Start reading file...');
const data = fs.readFileSync('large-file.txt', 'utf8');
console.log('File read complete');
console.log('Doing other work...');
// Other work waits for file to be read completely

NON-BLOCKING (Asynchronous) - Good for servers

console.log('Start reading file...');
fs.readFile('large-file.txt', 'utf8', (err, data) => {
    if (err) throw err;
    console.log('File read complete');
});
console.log('Doing other work...'); // Executes immediately

// Output:
// Start reading file...
// Doing other work...
// File read complete (when ready)

Real-world example: Handling multiple requests

const http = require('http');

http.createServer((req, res) => {
    // This can handle thousands of concurrent requests
    fs.readFile('data.json', (err, data) => {
        // Other requests aren't blocked while waiting
        res.end(data);
    });
}).listen(3000);

Process & Thread Model

// Single thread handles all requests
console.log(process.pid); // Process ID
console.log(process.version); // Node version
console.log(process.platform); // OS platform
console.log(process.arch); // CPU architecture

// CPU-intensive task blocks event loop (BAD)
function blockingOperation() {
    let sum = 0;
    for (let i = 0; i < 1e9; i++) {
        sum += i;
    }
    return sum;
}

// Better: Use Worker Threads for CPU-intensive tasks
const { Worker, isMainThread, parentPort } = require('worker_threads');

if (isMainThread) {
    // Main thread
    const worker = new Worker(__filename);
    worker.on('message', (result) => {
        console.log('Result from worker:', result);
    });
    worker.postMessage('start');
} else {
    // Worker thread
    parentPort.on('message', (msg) => {
        const result = blockingOperation();
        parentPort.postMessage(result);
    });
}

// Or use Child Processes
const { fork } = require('child_process');

const child = fork('child.js');
child.send({ task: 'heavy-computation' });
child.on('message', (result) => {
    console.log('Result:', result);
});

Cluster Module (Multi-core Usage)

const cluster = require('cluster');
const os = require('os');
const http = require('http');

if (cluster.isMaster) {
    const numCPUs = os.cpus().length;
    console.log(`Master process ${process.pid} is running`);

    // Fork workers
    for (let i = 0; i < numCPUs; i++) {
        cluster.fork();
    }

    cluster.on('exit', (worker, code, signal) => {
        console.log(`Worker ${worker.process.pid} died`);
        cluster.fork(); // Replace dead worker
    });
} else {
    // Worker process
    http.createServer((req, res) => {
        res.writeHead(200);
        res.end('Hello World\n');
    }).listen(8000);

    console.log(`Worker ${process.pid} started`);
}

4. Core Modules Complete Guide

File System (fs)

Definition: The fs module provides file system operations like reading, writing, and manipulating files and directories.

READING FILES

const fs = require('fs');
const path = require('path');

// Synchronous (blocks code execution)
try {
    const data = fs.readFileSync('file.txt', 'utf8');
    console.log(data);
} catch (err) {
    console.error(err);
}

// Asynchronous (non-blocking)
fs.readFile('file.txt', 'utf8', (err, data) => {
    if (err) {
        console.error(err);
        return;
    }
    console.log(data);
});

// Promises (modern approach)
const fsPromises = require('fs').promises;

async function readFileAsync() {
    try {
        const data = await fsPromises.readFile('file.txt', 'utf8');
        console.log(data);
    } catch (err) {
        console.error(err);
    }
}

WRITING FILES

// Write (overwrites existing file)
fs.writeFile('output.txt', 'Hello World', (err) => {
if (err) throw err;
console.log('File written');
});

// Append (adds to existing file)
fs.appendFile('log.txt', 'New log entry\n', (err) => {
if (err) throw err;
console.log('Log added');
});

// Write with options
const content = 'Important data';
fs.writeFile('data.txt', content, {
encoding: 'utf8',
mode: 0o666,
flag: 'w'
}, (err) => {
if (err) throw err;
});


**FILE OPERATIONS**

// Check if file exists
fs.access('file.txt', fs.constants.F_OK, (err) => {
if (err) {
console.log('File does not exist');
} else {
console.log('File exists');
}
});

// Get file stats
fs.stat('file.txt', (err, stats) => {
if (err) throw err;
console.log('File size:', stats.size);
console.log('Is file:', stats.isFile());
console.log('Is directory:', stats.isDirectory());
console.log('Modified:', stats.mtime);
});

// Rename file
fs.rename('old.txt', 'new.txt', (err) => {
if (err) throw err;
console.log('File renamed');
});

// Delete file
fs.unlink('file.txt', (err) => {
if (err) throw err;
console.log('File deleted');
});

// Copy file
fs.copyFile('source.txt', 'destination.txt', (err) => {
if (err) throw err;
console.log('File copied');
});


**DIRECTORY OPERATIONS**

// Create directory
fs.mkdir('new-folder', { recursive: true }, (err) => {
if (err) throw err;
console.log('Directory created');
});

// Read directory
fs.readdir('./', (err, files) => {
if (err) throw err;
console.log('Files:', files);

// Filter files by extension
const txtFiles = files.filter(f => path.extname(f) === '.txt');

});

// Read directory with file types
fs.readdir('./', { withFileTypes: true }, (err, entries) => {
if (err) throw err;
entries.forEach(entry => {
if (entry.isFile()) {
console.log('File:', entry.name);
} else if (entry.isDirectory()) {
console.log('Directory:', entry.name);
}
});
});

// Remove directory
fs.rmdir('folder', { recursive: true }, (err) => {
if (err) throw err;
console.log('Directory removed');
});


**WATCHING FILES**

// Watch for file changes
fs.watch('config.json', (eventType, filename) => {
console.log(Event: ${eventType});
if (filename) {
console.log(File changed: ${filename});
}
});

// Watch with more control
const watcher = fs.watch('watched-folder', { recursive: true }, (event, file) => {
console.log(${file} was ${event});
});

// Stop watching
setTimeout(() => {
watcher.close();
}, 10000);


**STREAMS (for large files)**

// Read stream (efficient for large files)
const readStream = fs.createReadStream('large-file.txt', {
encoding: 'utf8',
highWaterMark: 16 * 1024 // 16KB chunks
});

readStream.on('data', (chunk) => {
console.log('Received chunk:', chunk.length);
});

readStream.on('end', () => {
console.log('Finished reading');
});

readStream.on('error', (err) => {
console.error('Error:', err);
});

// Write stream
const writeStream = fs.createWriteStream('output.txt');

writeStream.write('First line\n');
writeStream.write('Second line\n');
writeStream.end('Last line\n');

writeStream.on('finish', () => {
console.log('Finished writing');
});

// Pipe streams (copy file efficiently)
const source = fs.createReadStream('source.txt');
const destination = fs.createWriteStream('destination.txt');

source.pipe(destination);

source.on('end', () => {
console.log('File copied');
});


**PRACTICAL EXAMPLES**

// Read JSON file
async function readJSON(filename) {
const data = await fsPromises.readFile(filename, 'utf8');
return JSON.parse(data);
}

// Write JSON file
async function writeJSON(filename, obj) {
const data = JSON.stringify(obj, null, 2);
await fsPromises.writeFile(filename, data);
}

// List all files recursively
async function listFilesRecursive(dir, files = []) {
const entries = await fsPromises.readdir(dir, { withFileTypes: true });

for (const entry of entries) {
    const fullPath = path.join(dir, entry.name);
    if (entry.isDirectory()) {
        await listFilesRecursive(fullPath, files);
    } else {
        files.push(fullPath);
    }
}

return files;

}

// Copy directory recursively
async function copyDir(src, dest) {
await fsPromises.mkdir(dest, { recursive: true });
const entries = await fsPromises.readdir(src, { withFileTypes: true });

for (const entry of entries) {
    const srcPath = path.join(src, entry.name);
    const destPath = path.join(dest, entry.name);

    if (entry.isDirectory()) {
        await copyDir(srcPath, destPath);
    } else {
        await fsPromises.copyFile(srcPath, destPath);
    }
}

}


### Path Module

const path = require('path');

// Join paths (cross-platform)
const filePath = path.join('/users', 'john', 'documents', 'file.txt');
// Linux/Mac: /users/john/documents/file.txt
// Windows: \users\john\documents\file.txt

// Resolve to absolute path
const absolutePath = path.resolve('docs', 'file.txt');
// /current/working/directory/docs/file.txt

path.resolve('/foo', '/bar', 'file.txt'); // /bar/file.txt
path.resolve('foo', 'bar', 'file.txt'); // /current/dir/foo/bar/file.txt

// Parse path
const parsed = path.parse('/home/user/docs/file.txt');
console.log(parsed);
// {
// root: '/',
// dir: '/home/user/docs',
// base: 'file.txt',
// ext: '.txt',
// name: 'file'
// }

// Build path from object
const filePath2 = path.format({
dir: '/home/user/docs',
base: 'file.txt'
});
// /home/user/docs/file.txt

// Get file extension
const ext = path.extname('file.txt'); // .txt
const ext2 = path.extname('archive.tar.gz'); // .gz

// Get filename
const base = path.basename('/home/user/file.txt'); // file.txt
const name = path.basename('/home/user/file.txt', '.txt'); // file

// Get directory name
const dir = path.dirname('/home/user/docs/file.txt');
// /home/user/docs

// Normalize path (resolve .. and .)
const normalized = path.normalize('/home/user/../user/./docs//file.txt');
// /home/user/docs/file.txt

// Check if path is absolute
const isAbs = path.isAbsolute('/home/user'); // true
const isAbs2 = path.isAbsolute('docs/file.txt'); // false

// Relative path between two paths
const rel = path.relative('/home/user/docs', '/home/user/photos/pic.jpg');
// ../photos/pic.jpg

// Path separator
console.log(path.sep); // '/' on Unix, '\' on Windows
console.log(path.delimiter); // ':' on Unix, ';' on Windows

// Practical examples
const filename = '/home/user/app/index.js';
const __dirname = path.dirname(filename); // /home/user/app

// Safe file paths
function getUploadPath(filename) {
// Prevent directory traversal attacks
const safeName = path.basename(filename);
return path.join(__dirname, 'uploads', safeName);
}

// ../../../etc/passwd becomes etc-passwd
const safe = getUploadPath('../../../etc/passwd');


### HTTP Module

const http = require('http');

// Create server
const server = http.createServer((req, res) => {
// Set response header
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');

// Send response
res.end('Hello World\n');

});

// Start server
server.listen(3000, '127.0.0.1', () => {
console.log('Server running at http://127.0.0.1:3000/');
});

// Routing
const server2 = http.createServer((req, res) => {
const { method, url } = req;

if (url === '/') {
    res.writeHead(200, { 'Content-Type': 'text/html' });
    res.end('<h1>Home Page</h1>');
} else if (url === '/about') {
    res.writeHead(200, { 'Content-Type': 'text/html' });
    res.end('<h1>About Page</h1>');
} else if (url === '/api/users' && method === 'GET') {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ users: ['John', 'Jane'] }));
} else {
    res.writeHead(404, { 'Content-Type': 'text/html' });
    res.end('<h1>404 Not Found</h1>');
}

});

// Handling POST data
const server3 = http.createServer((req, res) => {
if (req.method === 'POST' && req.url === '/api/data') {
let body = '';

    // Collect data chunks
    req.on('data', chunk => {
        body += chunk.toString();
    });

    // Process complete data
    req.on('end', () => {
        try {
            const data = JSON.parse(body);
            console.log('Received:', data);

            res.writeHead(200, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ message: 'Data received', data }));
        } catch (err) {
            res.writeHead(400, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ error: 'Invalid JSON' }));
        }
    });
}

});

// Query parameters
const url = require('url');

const server4 = http.createServer((req, res) => {
const parsedUrl = url.parse(req.url, true);
const { pathname, query } = parsedUrl;

if (pathname === '/search') {
    const searchTerm = query.q;
    const page = query.page || 1;

    res.end(`Searching for: ${searchTerm}, Page: ${page}`);
}

});

// Making HTTP requests
http.get('http://api.example.com/data', (res) => {
let data = '';

res.on('data', chunk => {
    data += chunk;
});

res.on('end', () => {
    console.log('Response:', data);
});

}).on('error', (err) => {
console.error('Error:', err.message);
});

// POST request
const options = {
hostname: 'api.example.com',
port: 80,
path: '/api/users',
method: 'POST',
headers: {
'Content-Type': 'application/json'
}
};

const req = http.request(options, (res) => {
console.log(Status: ${res.statusCode});

res.on('data', (chunk) => {
    console.log(`Body: ${chunk}`);
});

});

req.on('error', (err) => {
console.error('Error:', err);
});

req.write(JSON.stringify({ name: 'John', age: 30 }));
req.end();

**Events Module**
**Definition**: The Events module provides an EventEmitter class for handling custom events and callbacks.

const EventEmitter = require('events');

// Basic usage
class MyEmitter extends EventEmitter {}

const myEmitter = new MyEmitter();

// Register event listener
myEmitter.on('event', (arg1, arg2) => {
console.log('Event fired!', arg1, arg2);
});

// Emit event
myEmitter.emit('event', 'Hello', 'World');
// Output: Event fired! Hello World

// One-time listeners
myEmitter.once('oneTime', () => {
console.log('This runs only once');
});

myEmitter.emit('oneTime'); // Runs
myEmitter.emit('oneTime'); // Doesn't run

// Multiple listeners
myEmitter.on('multi', () => console.log('Listener 1'));
myEmitter.on('multi', () => console.log('Listener 2'));
myEmitter.on('multi', () => console.log('Listener 3'));

myEmitter.emit('multi');
// Output:
// Listener 1
// Listener 2
// Listener 3

// Removing listeners
function listener1() {
console.log('Listener 1');
}

function listener2() {
console.log('Listener 2');
}

myEmitter.on('test', listener1);
myEmitter.on('test', listener2);

// Remove specific listener
myEmitter.removeListener('test', listener1);

// Remove all listeners for event
myEmitter.removeAllListeners('test');

// Error handling
myEmitter.on('error', (err) => {
console.error('Error occurred:', err.message);
});

myEmitter.emit('error', new Error('Something went wrong'));

// Practical example: User Service
class UserService extends EventEmitter {
createUser(userData) {
// Validate user
if (!userData.email) {
this.emit('error', new Error('Email required'));
return;
}

    // Create user
    const user = {
        id: Date.now(),
        ...userData,
        createdAt: new Date()
    };

    // Emit success event
    this.emit('userCreated', user);

    return user;
}

deleteUser(userId) {
    this.emit('userDeleted', userId);
}

}

const userService = new UserService();

// Listen to events
userService.on('userCreated', (user) => {
console.log('New user:', user);
// Send welcome email
// Log to database
// Update analytics
});

userService.on('userDeleted', (userId) => {
console.log('User deleted:', userId);
// Cleanup user data
// Send notification
});

userService.on('error', (err) => {
console.error('User service error:', err.message);
});

// Use the service
userService.createUser({ email: 'john@example.com', name: 'John' });
userService.createUser({ name: 'Jane' }); // Triggers error

**Buffer Module**
**Definition**: Buffers are used to handle binary data in Node.js. They are like arrays of bytes.

// Creating buffers
// From string
const buf1 = Buffer.from('Hello World');
const buf2 = Buffer.from('Hello', 'utf8');
const buf3 = Buffer.from([72, 101, 108, 108, 111]); // ASCII codes

// Allocate buffer (filled with zeros)
const buf4 = Buffer.alloc(10); // Safe, initialized
const buf5 = Buffer.allocUnsafe(10); // Faster, not initialized

// Fill buffer
const buf6 = Buffer.alloc(10, 'a'); // Fill with 'a'

// Reading buffers
console.log(buf1.toString()); // Hello World
console.log(buf1.toString('hex')); // 48656c6c6f20576f726c64
console.log(buf1.toString('base64')); // SGVsbG8gV29ybGQ=
console.log(buf1.toString('utf8', 0, 5)); // Hello

// Read specific bytes
console.log(buf1[0]); // 72 (ASCII for 'H')
console.log(buf1.length); // 11

// Writing to buffers
const buf7 = Buffer.alloc(10);

// Write string
buf7.write('Hello');
console.log(buf7.toString()); // Hello

// Write at offset
buf7.write('Hi', 0);
console.log(buf7.toString()); // Hillo

// Buffer operations
// Copy buffer
const source = Buffer.from('Hello');
const target = Buffer.alloc(5);
source.copy(target);
console.log(target.toString()); // Hello

// Slice buffer (shares memory)
const buf8 = Buffer.from('Hello World');
const slice = buf8.slice(0, 5);
console.log(slice.toString()); // Hello

// Concat buffers
const buf9 = Buffer.from('Hello ');
const buf10 = Buffer.from('World');
const concat = Buffer.concat([buf9, buf10]);
console.log(concat.toString()); // Hello World

// Compare buffers
const buf11 = Buffer.from('ABC');
const buf12 = Buffer.from('BCD');
console.log(buf11.compare(buf12)); // -1 (buf11 < buf12)
console.log(buf11.equals(buf12)); // false

// Fill buffer
const buf13 = Buffer.alloc(10);
buf13.fill('a');
console.log(buf13.toString()); // aaaaaaaaaa

// Encoding/Decoding
// Base64 encoding
const original = 'Hello World';
const encoded = Buffer.from(original).toString('base64');
const decoded = Buffer.from(encoded, 'base64').toString('utf8');
console.log(encoded); // SGVsbG8gV29ybGQ=
console.log(decoded); // Hello World

// Hex encoding
const hex = Buffer.from('Hello').toString('hex');
console.log(hex); // 48656c6c6f
const fromHex = Buffer.from(hex, 'hex').toString();
console.log(fromHex); // Hello

// Practical examples
// Read binary file
const fs = require('fs');

fs.readFile('image.png', (err, buffer) => {
if (err) throw err;
console.log('File size:', buffer.length);
console.log('First byte:', buffer[0]);

// Convert to base64 for web
const base64 = buffer.toString('base64');
const dataUrl = `data:image/png;base64,${base64}`;

});

// Generate random buffer
const crypto = require('crypto');
const randomBuffer = crypto.randomBytes(16);
console.log(randomBuffer.toString('hex'));

// Create hash
const hash = crypto.createHash('sha256');
hash.update('password123');
console.log(hash.digest('hex'));

// Type checking
console.log(Buffer.isBuffer(buf1)); // true
console.log(Buffer.isBuffer('string')); // false

// Check encoding support
console.log(Buffer.isEncoding('utf8')); // true
console.log(Buffer.isEncoding('utf-99')); // false

// Get byte length
console.log(Buffer.byteLength('Hello')); // 5
console.log(Buffer.byteLength('こんにちは')); // 15 (UTF-8)

---

## 5. NPM & Package Management

### What is NPM?

**Definition:** NPM (Node Package Manager) is the default package manager for Node.js. It's the world's largest software registry with over 2 million packages.

### Basic Commands

Check version

npm --version
npm -v

Initialize project

npm init # Interactive
npm init -y # Default values

Install package locally

npm install express
npm i express # Short form
npm install express@4.17.1 # Specific version

Install as dev dependency

npm install --save-dev nodemon
npm install -D nodemon

Install globally

npm install -g typescript
npm i -g typescript

Uninstall package

npm uninstall express
npm remove express
npm rm express

Update packages

npm update # Update all
npm update express # Update specific

Outdated packages

npm outdated

List installed packages

npm list
npm list --depth=0 # Top-level only
npm list -g # Global packages

View package info

npm view express
npm info express

Search packages

npm search mongodb

Audit for vulnerabilities

npm audit
npm audit fix # Fix automatically
npm audit fix --force # Force fix (breaking changes)

Clean cache

npm cache clean --force

Run scripts

npm start
npm test
npm run dev
npm run build

Prune unused packages

npm prune
npm prune --production # Remove devDependencies

Link local package (development)

npm link
npm link package-name

Publish package

npm publish
npm unpublish package-name@version


### package.json Explained

{
"name": "my-app",
"version": "1.0.0",
"description": "My awesome application",
"main": "index.js",
"scripts": {
"start": "node index.js",
"dev": "nodemon index.js",
"test": "jest",
"build": "webpack",
"lint": "eslint .",
"format": "prettier --write ."
},
"keywords": ["api", "backend", "nodejs"],
"author": "Your Name your.email@example.com",
"license": "MIT",
"dependencies": {
"express": "^4.18.2",
"mongoose": "^7.0.0",
"dotenv": "^16.0.3"
},
"devDependencies": {
"nodemon": "^2.0.20",
"jest": "^29.0.0",
"eslint": "^8.30.0"
},
"engines": {
"node": ">=14.0.0",
"npm": ">=6.0.0"
},
"repository": {
"type": "git",
"url": "https://github.com/username/repo.git"
},
"bugs": {
"url": "https://github.com/username/repo/issues"
},
"homepage": "https://github.com/username/repo#readme"
}


### Version Numbers Explained

Major.Minor.Patch
2 . 3 . 5

Major: Breaking changes
Minor: New features (backward compatible)
Patch: Bug fixes

Symbols:
^2.3.5 - Compatible with 2.x.x (>= 2.3.5, < 3.0.0)
~2.3.5 - Approximately 2.3.x (>= 2.3.5, < 2.4.0)
2.3.5 - Exact version

=2.3.5 - Greater than or equal

- Any version latest - Latest version


### NPM Scripts

{
"scripts": {
"start": "node server.js",
"dev": "nodemon server.js",
"test": "jest --coverage",
"test:watch": "jest --watch",
"build": "webpack --mode production",
"clean": "rm -rf dist",
"prebuild": "npm run clean",
"postbuild": "echo 'Build complete'",
"deploy": "npm run build && npm run upload",
"lint": "eslint src//*.js",
"format": "prettier --write src//*.js",
"prepare": "husky install",
"custom": "node scripts/custom.js"
}
}

Run scripts

npm start # Built-in, no 'run' needed
npm test # Built-in
npm run dev # Custom scripts need 'run'
npm run build

Pre and post hooks

npm run build # Runs: prebuild → build → postbuild

Pass arguments

npm start -- --port=3000
npm test -- --watch


### package-lock.json

**Purpose:**
- Locks exact version of all dependencies
- Ensures consistent installs across machines
- Improves install speed
- Should be committed to version control

{
"name": "my-app",
"version": "1.0.0",
"lockfileVersion": 2,
"requires": true,
"packages": {
"": {
"name": "my-app",
"version": "1.0.0",
"dependencies": {
"express": "^4.18.2"
}
},
"node_modules/express": {
"version": "4.18.2",
"resolved": "https://registry.npmjs.org/express/-/express-4.18.2.tgz",
"integrity": "sha512-...",
"dependencies": {
"body-parser": "1.20.1"
}
}
}
}


### .npmrc Configuration

.npmrc file

Set registry

registry=https://registry.npmjs.org/

Private registry

@mycompany:registry=https://npm.mycompany.com/

Always save exact versions

save-exact=true

Install devDependencies by default

save-dev=true

Set cache directory

cache=/path/to/cache

Proxy settings

proxy=http://proxy.company.com:8080
https-proxy=http://proxy.company.com:8080

Authentication token

//registry.npmjs.org/:_authToken=YOUR_TOKEN

Log level

loglevel=silent # or error, warn, info, verbose, silly


### .npmignore

.npmignore file (files to exclude from published package)

Source files

src/
tests/
*.test.js

Build tools

webpack.config.js
.babelrc
.eslintrc

Documentation

docs/
*.md
!README.md

Environment

.env
.env.local

Dependencies

node_modules/

IDE

.vscode/
.idea/

Version control

.git/
.gitignore


### Common Packages

Web frameworks

npm install express
npm install fastify
npm install koa

Database

npm install mongoose # MongoDB
npm install pg # PostgreSQL
npm install mysql2 # MySQL
npm install redis

Authentication

npm install jsonwebtoken
npm install bcrypt
npm install passport

Validation

npm install joi
npm install yup
npm install validator
npm install express-validator

Environment variables

npm install dotenv

HTTP client

npm install axios
npm install node-fetch

Utilities

npm install lodash
npm install moment # Date manipulation
npm install uuid # Generate UUIDs

Testing

npm install --save-dev jest
npm install --save-dev mocha chai
npm install --save-dev supertest

Linting & Formatting

npm install --save-dev eslint
npm install --save-dev prettier

Process management

npm install --save-dev nodemon
npm install -g pm2

File upload

npm install multer

Template engines

npm install ejs
npm install pug
npm install handlebars

Logging

npm install winston
npm install morgan

CORS

npm install cors

Rate limiting

npm install express-rate-limit

Compression

npm install compression

Security

npm install helmet
npm install express-mongo-sanitize
npm install xss-clean

WebSockets

npm install socket.io

Queue

npm install bull

Email

npm install nodemailer

GraphQL

npm install apollo-server-express graphql

Documentation

npm install swagger-ui-express


### Creating Your Own Package

1. Initialize package

mkdir my-package
cd my-package
npm init

2. Create index.js

// index.js
function greet(name) {
return Hello, ${name}!;
}

function add(a, b) {
return a + b;
}

module.exports = {
greet,
add
};

3. Test locally

npm link

4. In another project

npm link my-package

5. Use it

const { greet, add } = require('my-package');

console.log(greet('World')); // Hello, World!
console.log(add(2, 3)); // 5

6. Publish to NPM

npm login
npm publish

Update version

npm version patch # 1.0.0 → 1.0.1
npm version minor # 1.0.0 → 1.1.0
npm version major # 1.0.0 → 2.0.0
npm publish


### Troubleshooting

Permission errors (avoid sudo)

npm config get prefix # Check prefix
mkdir ~/.npm-global
npm config set prefix '~/.npm-global'

Add to PATH: export PATH=~/.npm-global/bin:$PATH

Clear cache and reinstall

rm -rf node_modules package-lock.json
npm cache clean --force
npm install

Fix vulnerabilities

npm audit
npm audit fix
npm audit fix --force

Check for updates

npx npm-check-updates
npx npm-check-updates -u # Update package.json

Verify installation

npm doctor

Debug installation

npm install --verbose
npm install --loglevel silly


---

## 6. Express.js Framework 

### What is Express?

**Definition:** Express.js is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications.

### Basic Setup

const express = require('express');
const app = express();

// Start server
app.listen(3000, () => {
console.log('Server running on port 3000');
});


### Middleware

**Definition:** Middleware functions have access to the request object (req), response object (res), and the next middleware function in the application's request-response cycle.


**BUILT-IN MIDDLEWARE**

// Parse JSON bodies
app.use(express.json());

// Parse URL-encoded bodies
app.use(express.urlencoded({ extended: true }));

// Serve static files
app.use(express.static('public'));
app.use('/static', express.static('public'));


**CUSTOM MIDDLEWARE**

// Logger middleware
app.use((req, res, next) => {
console.log(${req.method} ${req.url} - ${new Date().toISOString()});
next(); // Pass control to next middleware
});

// Authentication middleware
const authenticate = (req, res, next) => {
const token = req.headers['authorization'];

if (!token) {
    return res.status(401).json({ error: 'No token provided' });
}

// Verify token (simplified)
if (token === 'valid-token') {
    req.user = { id: 1, name: 'John' };
    next();
} else {
    res.status(403).json({ error: 'Invalid token' });
}

};

// Apply to specific routes
app.get('/protected', authenticate, (req, res) => {
res.json({ message: 'Protected data', user: req.user });
});


**ERROR HANDLING MIDDLEWARE**

// Must have 4 parameters
app.use((err, req, res, next) => {
console.error(err.stack);
res.status(err.status || 500).json({
error: {
message: err.message || 'Internal Server Error'
}
});
});


**MIDDLEWARE ORDER**

// This won't work (middleware after routes)
app.get('/user', (req, res) => {
res.json(req.body); // undefined
});
app.use(express.json()); // Too late!

// Correct order
app.use(express.json());
app.get('/user', (req, res) => {
res.json(req.body); // Works!
});


**CONDITIONAL MIDDLEWARE**

app.use('/api', (req, res, next) => {
console.log('API route accessed');
next();
});


**THIRD-PARTY MIDDLEWARE**

const morgan = require('morgan');
const cors = require('cors');
const helmet = require('helmet');
const compression = require('compression');

app.use(morgan('dev')); // Logging
app.use(cors()); // CORS
app.use(helmet()); // Security headers
app.use(compression()); // Response compression


**ROUTING**
Basic Routes

// GET request
app.get('/', (req, res) => {
res.send('Hello World');
});

// POST request
app.post('/users', (req, res) => {
const user = req.body;
res.status(201).json({ user });
});

// PUT request
app.put('/users/:id', (req, res) => {
const id = req.params.id;
res.json({ message: Update user ${id} });
});

// DELETE request
app.delete('/users/:id', (req, res) => {
const id = req.params.id;
res.json({ message: Delete user ${id} });
});


Route Parameters

// Single parameter
app.get('/users/:id', (req, res) => {
const userId = req.params.id;
res.send(User ID: ${userId});
});

// Multiple parameters
app.get('/posts/:year/:month/:day', (req, res) => {
const { year, month, day } = req.params;
res.send(Date: ${year}-${month}-${day});
});

// Optional parameters
app.get('/users/:id/:name?', (req, res) => {
res.json(req.params);
});

// Regular expressions
app.get('/products/:id(\d+)', (req, res) => {
// Only matches numeric IDs
res.send(Product ID: ${req.params.id});
});


**QUERY PARAMETERS**

app.get('/search', (req, res) => {
const { q, page = 1, limit = 10 } = req.query;
res.json({
query: q,
page: parseInt(page),
limit: parseInt(limit)
});
});
// GET /search?q=nodejs&page=2&limit=20


**ROUTER**

// Create router instance
const userRouter = express.Router();

// Define routes
userRouter.get('/', (req, res) => {
res.json({ message: 'Get all users' });
});

userRouter.get('/:id', (req, res) => {
res.json({ message: Get user ${req.params.id} });
});

userRouter.post('/', (req, res) => {
res.json({ message: 'Create user' });
});

userRouter.put('/:id', (req, res) => {
res.json({ message: Update user ${req.params.id} });
});

userRouter.delete('/:id', (req, res) => {
res.json({ message: Delete user ${req.params.id} });
});

// Mount router
app.use('/api/users', userRouter);


**ROUTE CHAINING**

app.route('/book')
.get((req, res) => {
res.send('Get all books');
})
.post((req, res) => {
res.send('Add a book');
})
.put((req, res) => {
res.send('Update a book');
});


**ORGANIZED ROUTES**

// routes/users.js
const express = require('express');
const router = express.Router();

router.get('/', getAllUsers);
router.get('/:id', getUserById);
router.post('/', createUser);
router.put('/:id', updateUser);
router.delete('/:id', deleteUser);

module.exports = router;

// app.js
const userRoutes = require('./routes/users');
app.use('/api/users', userRoutes);

Request & Response Objects

REQUEST OBJECT

app.post('/test', (req, res) => {
    // Request properties
    console.log(req.method);        // POST
    console.log(req.url);           // /test
    console.log(req.path);          // /test
    console.log(req.hostname);      // example.com
    console.log(req.protocol);      // http or https
    console.log(req.ip);            // Client IP address

    // Headers
    console.log(req.headers);
    console.log(req.get('Content-Type'));

    // Body (requires middleware)
    console.log(req.body);

    // Query parameters
    console.log(req.query);

    // Route parameters
    console.log(req.params);

    // Cookies (requires cookie-parser)
    console.log(req.cookies);

    // Check content type
    if (req.is('json')) {
        console.log('JSON request');
    }
});

RESPONSE OBJECT

app.get('/response', (req, res) => {
    // Send text
    res.send('Hello World');

    // Send JSON
    res.json({ message: 'Success', data: [] });

    // Send status code
    res.status(404).send('Not Found');
    res.sendStatus(200); // Sends status text

    // Redirect
    res.redirect('/home');
    res.redirect(301, '/new-url');

    // Set headers
    res.set('Content-Type', 'text/html');
    res.header('X-Custom-Header', 'value');

    // Set multiple headers
    res.set({
        'Content-Type': 'text/plain',
        'X-Custom': 'value'
    });

    // Send file
    res.sendFile('/path/to/file.pdf');

    // Download file
    res.download('/path/to/file.pdf');
    res.download('/path/to/file.pdf', 'custom-name.pdf');

    // Render template
    res.render('index', { title: 'Home' });

    // Chain methods
    res.status(201).json({ message: 'Created' });
});

COOKIES

app.get('/set-cookie', (req, res) => {
    // Simple cookie
    res.cookie('name', 'value');

    // Cookie with options
    res.cookie('token', 'abc123', {
        maxAge: 900000,        // 15 minutes
        httpOnly: true,        // Not accessible via JavaScript
        secure: true,          // HTTPS only
        signed: true,          // Signed cookie
        sameSite: 'strict'     // CSRF protection
    });

    res.send('Cookie set');
});

app.get('/clear-cookie', (req, res) => {
    res.clearCookie('name');
    res.send('Cookie cleared');
});

Template Engines

EJS

// Install: npm install ejs

// Setup
app.set('view engine', 'ejs');
app.set('views', './views');

// views/index.ejs
/*
<!DOCTYPE html>
<html>
<head>
    <title><%= title %></title>
</head>
<body>
    <h1>Welcome <%= user.name %></h1>
    <% if (users.length > 0) { %>
        <ul>
            <% users.forEach(user => { %>
                <li><%= user.name %></li>
            <% }); %>
        </ul>
    <% } else { %>
        <p>No users found</p>
    <% } %>
    <%- include('partials/header') %>
</body>
</html>
*/

// Route
app.get('/', (req, res) => {
    res.render('index', {
        title: 'Home Page',
        user: { name: 'John' },
        users: [
            { name: 'Alice' },
            { name: 'Bob' }
        ]
    });
});

PUG

// Install: npm install pug

// Setup
app.set('view engine', 'pug');

// views/index.pug
/*
doctype html
html
  head
    title= title
  body
    h1 Welcome #{user.name}

    if users.length > 0
      ul
        each user in users
          li= user.name
    else
      p No users found

    include partials/header
*/

// Route
app.get('/', (req, res) => {
    res.render('index', {
        title: 'Home Page',
        user: { name: 'John' },
        users: [{ name: 'Alice' }, { name: 'Bob' }]
    });
});

7. REST API Design

What is REST?

Definition: REST (Representational State Transfer) is an architectural style for designing networked applications. It uses HTTP methods to perform CRUD operations on resources.

REST Principles:

Client-Server: Separation of concerns
Stateless: Each request contains all information needed
Cacheable: Responses can be cached
Uniform Interface: Standardized communication
Layered System: Client doesn't know if connected to end server

HTTP Methods

GET     - Retrieve data (Read)
POST    - Create new data (Create)
PUT     - Update entire resource (Update/Replace)
PATCH   - Update part of resource (Partial Update)
DELETE  - Remove data (Delete)
HEAD    - Get headers only
OPTIONS - Get allowed methods

RESTful Routes

const express = require('express');
const app = express();

app.use(express.json());

// Sample data
let users = [
    { id: 1, name: 'John Doe', email: 'john@example.com' },
    { id: 2, name: 'Jane Smith', email: 'jane@example.com' }
];

// GET - Retrieve all resources
app.get('/api/users', (req, res) => {
    res.json({
        success: true,
        data: users,
        count: users.length
    });
});

// GET - Retrieve single resource
app.get('/api/users/:id', (req, res) => {
    const user = users.find(u => u.id === parseInt(req.params.id));

    if (!user) {
        return res.status(404).json({
            success: false,
            error: 'User not found'
        });
    }

    res.json({
        success: true,
        data: user
    });
});

// POST - Create new resource
app.post('/api/users', (req, res) => {
    const { name, email } = req.body;

    // Validation
    if (!name || !email) {
        return res.status(400).json({
            success: false,
            error: 'Name and email are required'
        });
    }

    const newUser = {
        id: users.length + 1,
        name,
        email
    };

    users.push(newUser);

    res.status(201).json({
        success: true,
        data: newUser
    });
});

// PUT - Update entire resource
app.put('/api/users/:id', (req, res) => {
    const user = users.find(u => u.id === parseInt(req.params.id));

    if (!user) {
        return res.status(404).json({
            success: false,
            error: 'User not found'
        });
    }

    const { name, email } = req.body;

    if (!name || !email) {
        return res.status(400).json({
            success: false,
            error: 'Name and email are required'
        });
    }

    user.name = name;
    user.email = email;

    res.json({
        success: true,
        data: user
    });
});

// PATCH - Update partial resource
app.patch('/api/users/:id', (req, res) => {
    const user = users.find(u => u.id === parseInt(req.params.id));

    if (!user) {
        return res.status(404).json({
            success: false,
            error: 'User not found'
        });
    }

    const { name, email } = req.body;

    if (name) user.name = name;
    if (email) user.email = email;

    res.json({
        success: true,
        data: user
    });
});

// DELETE - Remove resource
app.delete('/api/users/:id', (req, res) => {
    const index = users.findIndex(u => u.id === parseInt(req.params.id));

    if (index === -1) {
        return res.status(404).json({
            success: false,
            error: 'User not found'
        });
    }

    users.splice(index, 1);

    res.json({
        success: true,
        message: 'User deleted'
    });
});

API Best Practices

VERSIONING

// URL versioning (recommended)
app.use('/api/v1/users', userRoutesV1);
app.use('/api/v2/users', userRoutesV2);

// Header versioning
app.use((req, res, next) => {
    const version = req.get('API-Version') || '1';
    req.apiVersion = version;
    next();
});

PAGINATION

app.get('/api/users', (req, res) => {
    const page = parseInt(req.query.page) || 1;
    const limit = parseInt(req.query.limit) || 10;
    const startIndex = (page - 1) * limit;
    const endIndex = page * limit;

    const results = {};

    if (endIndex < users.length) {
        results.next = {
            page: page + 1,
            limit
        };
    }

    if (startIndex > 0) {
        results.previous = {
            page: page - 1,
            limit
        };
    }

    results.data = users.slice(startIndex, endIndex);
    results.total = users.length;
    results.pages = Math.ceil(users.length / limit);

    res.json(results);
});

FILTERING

app.get('/api/users', (req, res) => {
    let filteredUsers = [...users];

    // Filter by name
    if (req.query.name) {
        filteredUsers = filteredUsers.filter(u =>
            u.name.toLowerCase().includes(req.query.name.toLowerCase())
        );
    }

    // Filter by email
    if (req.query.email) {
        filteredUsers = filteredUsers.filter(u =>
            u.email.toLowerCase().includes(req.query.email.toLowerCase())
        );
    }

    res.json({
        success: true,
        count: filteredUsers.length,
        data: filteredUsers
    });
});

SORTING

app.get('/api/users', (req, res) => {
    let sortedUsers = [...users];

    // Sort by field
    const sortBy = req.query.sortBy || 'id';
    const order = req.query.order === 'desc' ? -1 : 1;

    sortedUsers.sort((a, b) => {
        if (a[sortBy] < b[sortBy]) return -1 * order;
        if (a[sortBy] > b[sortBy]) return 1 * order;
        return 0;
    });

    res.json({
        success: true,
        data: sortedUsers
    });
});

FIELD SELECTION

app.get('/api/users', (req, res) => {
    let result = users;

    // Select specific fields
    if (req.query.fields) {
        const fields = req.query.fields.split(',');
        result = users.map(user => {
            const selected = {};
            fields.forEach(field => {
                if (user[field] !== undefined) {
                    selected[field] = user[field];
                }
            });
            return selected;
        });
    }

    res.json({
        success: true,
        data: result
    });
});
// GET /api/users?fields=name,email

SEARCHING

app.get('/api/users/search', (req, res) => {
    const searchTerm = req.query.q?.toLowerCase();

    if (!searchTerm) {
        return res.status(400).json({
            success: false,
            error: 'Search term required'
        });
    }

    const results = users.filter(u =>
        u.name.toLowerCase().includes(searchTerm) ||
        u.email.toLowerCase().includes(searchTerm)
    );

    res.json({
        success: true,
        count: results.length,
        data: results
    });
});

STANDARD RESPONSE FORMAT

const sendSuccess = (res, data, message = 'Success', statusCode = 200) => {
    res.status(statusCode).json({
        success: true,
        message,
        data
    });
};

const sendError = (res, error, statusCode = 500) => {
    res.status(statusCode).json({
        success: false,
        error: error.message || error
    });
};

// Usage
app.get('/api/users', (req, res) => {
    sendSuccess(res, users, 'Users retrieved successfully');
});

app.post('/api/users', (req, res) => {
    // Create user logic
    sendSuccess(res, newUser, 'User created successfully', 201);
});

HATEOAS (Hypermedia)

app.get('/api/users/:id', (req, res) => {
    const user = users.find(u => u.id === parseInt(req.params.id));

    if (!user) {
        return res.status(404).json({ error: 'User not found' });
    }

    res.json({
        data: user,
        links: {
            self: `/api/users/${user.id}`,
            update: {
                href: `/api/users/${user.id}`,
                method: 'PUT'
            },
            delete: {
                href: `/api/users/${user.id}`,
                method: 'DELETE'
            },
            posts: `/api/users/${user.id}/posts`
        }
    });
});

Status Codes

// ===== 2XX SUCCESS =====
200 - OK                    // Successful GET, PUT, PATCH
201 - Created              // Successful POST
204 - No Content           // Successful DELETE

// ===== 3XX REDIRECTION =====
301 - Moved Permanently    // Resource moved
302 - Found               // Temporary redirect
304 - Not Modified        // Cached version is valid

// ===== 4XX CLIENT ERRORS =====
400 - Bad Request         // Invalid input
401 - Unauthorized        // Not authenticated
403 - Forbidden           // Authenticated but no permission
404 - Not Found           // Resource doesn't exist
405 - Method Not Allowed  // HTTP method not supported
409 - Conflict            // Resource conflict
422 - Unprocessable       // Validation failed
429 - Too Many Requests   // Rate limit exceeded

// ===== 5XX SERVER ERRORS =====
500 - Internal Server Error  // Server error
502 - Bad Gateway           // Invalid response from upstream
503 - Service Unavailable   // Server temporarily down
504 - Gateway Timeout       // Upstream timeout

// Usage examples
app.post('/api/users', (req, res) => {
    if (!req.body.email) {
        return res.status(400).json({ error: 'Email required' });
    }

    // Create user
    res.status(201).json({ data: newUser });
});

app.get('/api/users/:id', (req, res) => {
    const user = findUser(req.params.id);

    if (!user) {
        return res.status(404).json({ error: 'User not found' });
    }

    res.status(200).json({ data: user });
});

app.delete('/api/users/:id', (req, res) => {
    // Delete user
    res.status(204).send();
});

8. Database Integration

MongoDB with Mongoose

Definition: MongoDB is a NoSQL document database. Mongoose is an ODM (Object Data Modeling) library for MongoDB and Node.js.

npm install mongoose

** CONNECTION**

const mongoose = require('mongoose');

// Connect to MongoDB
mongoose.connect('mongodb://localhost:27017/myapp', {
    useNewUrlParser: true,
    useUnifiedTopology: true
})
.then(() => console.log('MongoDB connected'))
.catch(err => console.error('MongoDB connection error:', err));

// With environment variable
const DB_URI = process.env.MONGODB_URI || 'mongodb://localhost:27017/myapp';
mongoose.connect(DB_URI);

// Connection events
mongoose.connection.on('connected', () => {
    console.log('Mongoose connected to DB');
});

mongoose.connection.on('error', (err) => {
    console.log('Mongoose connection error:', err);
});

mongoose.connection.on('disconnected', () => {
    console.log('Mongoose disconnected');
});

// Graceful shutdown
process.on('SIGINT', async () => {
    await mongoose.connection.close();
    process.exit(0);
});

SCHEMA DEFINITION

const userSchema = new mongoose.Schema({
    name: {
        type: String,
        required: [true, 'Name is required'],
        trim: true,
        minlength: [3, 'Name must be at least 3 characters'],
        maxlength: [50, 'Name cannot exceed 50 characters']
    },
    email: {
        type: String,
        required: true,
        unique: true,
        lowercase: true,
        trim: true,
        match: [/^\S+@\S+\.\S+$/, 'Invalid email format']
    },
    password: {
        type: String,
        required: true,
        minlength: 6,
        select: false  // Don't include in queries by default
    },
    age: {
        type: Number,
        min: [18, 'Must be at least 18'],
        max: 100
    },
    role: {
        type: String,
        enum: ['user', 'admin', 'moderator'],
        default: 'user'
    },
    isActive: {
        type: Boolean,
        default: true
    },
    avatar: String,
    tags: [String],
    address: {
        street: String,
        city: String,
        country: String,
        zipCode: String
    },
    socialLinks: {
        twitter: String,
        linkedin: String,
        github: String
    }
}, {
    timestamps: true  // Adds createdAt and updatedAt
});

VIRTUAL PROPERTIES

userSchema.virtual('fullInfo').get(function() {
    return `${this.name} (${this.email})`;
});

** INSTANCE METHODS **

userSchema.methods.comparePassword = async function(candidatePassword) {
    const bcrypt = require('bcrypt');
    return await bcrypt.compare(candidatePassword, this.password);
};

userSchema.methods.generateAuthToken = function() {
    const jwt = require('jsonwebtoken');
    return jwt.sign({ id: this._id }, process.env.JWT_SECRET);
};

** STATIC METHODS**

userSchema.statics.findByEmail = function(email) {
    return this.findOne({ email: email.toLowerCase() });
};

userSchema.statics.findActive = function() {
    return this.find({ isActive: true });
};

** MIDDLEWARE (HOOKS)**

// Pre-save hook
userSchema.pre('save', async function(next) {
    if (!this.isModified('password')) return next();

    const bcrypt = require('bcrypt');
    this.password = await bcrypt.hash(this.password, 10);
    next();
});

// Post-save hook
userSchema.post('save', function(doc, next) {
    console.log('User saved:', doc.name);
    next();
});

// Pre-remove hook
userSchema.pre('remove', function(next) {
    console.log('Removing user:', this.name);
    next();
});

** CREATE MODEL**

const User = mongoose.model('User', userSchema);

** CRUD OPERATIONS**

// CREATE
async function createUser() {
    const user = new User({
        name: 'John Doe',
        email: 'john@example.com',
        password: 'password123',
        age: 30,
        tags: ['developer', 'nodejs']
    });

    try {
        const savedUser = await user.save();
        console.log('User created:', savedUser);
        return savedUser;
    } catch (error) {
        console.error('Error creating user:', error.message);
    }
}

// Alternative create
async function createUser2() {
    try {
        const user = await User.create({
            name: 'Jane Smith',
            email: 'jane@example.com',
            password: 'password123'
        });
        console.log('User created:', user);
        return user;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

// READ - Find all
async function getAllUsers() {
    try {
        const users = await User.find();
        console.log('All users:', users);
        return users;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

// READ - Find with conditions
async function findUsers() {
    try {
        // Find users older than 25
        const users = await User.find({ age: { $gt: 25 } });

        // Find by multiple conditions
        const activeAdults = await User.find({
            age: { $gte: 18 },
            isActive: true
        });

        // Find with OR condition
        const results = await User.find({
            $or: [
                { role: 'admin' },
                { age: { $gt: 30 } }
            ]
        });

        return users;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

// READ - Find one
async function findUserById(id) {
    try {
        const user = await User.findById(id);
        console.log('User found:', user);
        return user;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

async function findUserByEmail(email) {
    try {
        const user = await User.findOne({ email });
        return user;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

// UPDATE
async function updateUser(id) {
    try {
        // Find and update
        const user = await User.findByIdAndUpdate(
            id,
            { name: 'Updated Name', age: 31 },
            { new: true, runValidators: true }
        );
        // new: true returns updated document
        // runValidators: true runs schema validators

        console.log('Updated user:', user);
        return user;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

async function updateUser2(id) {
    try {
        // Alternative: Find, modify, save
        const user = await User.findById(id);
        if (!user) throw new Error('User not found');

        user.name = 'New Name';
        user.age = 32;
        await user.save();

        return user;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

// Update multiple documents
async function updateManyUsers() {
    try {
        const result = await User.updateMany(
            { isActive: false },
            { isActive: true }
        );
        console.log(`Updated ${result.nModified} users`);
    } catch (error) {
        console.error('Error:', error.message);
    }
}

// DELETE
async function deleteUser(id) {
    try {
        const user = await User.findByIdAndDelete(id);
        console.log('Deleted user:', user);
        return user;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

// Delete multiple
async function deleteInactiveUsers() {
    try {
        const result = await User.deleteMany({ isActive: false });
        console.log(`Deleted ${result.deletedCount} users`);
    } catch (error) {
        console.error('Error:', error.message);
    }
}

QUERY METHODS

async function advancedQueries() {
    try {
        // Select specific fields
        const users = await User.find().select('name email -_id');

        // Limit results
        const limitedUsers = await User.find().limit(10);

        // Skip (for pagination)
        const page2 = await User.find().skip(10).limit(10);

        // Sort
        const sortedUsers = await User.find().sort({ age: -1 }); // Descending
        const sorted2 = await User.find().sort('name -age'); // name asc, age desc

        // Count
        const count = await User.countDocuments({ isActive: true });

        // Exists
        const usersWithAvatar = await User.find({ avatar: { $exists: true } });

        // Comparison operators
        const adults = await User.find({ age: { $gte: 18 } });
        const youngAdults = await User.find({ age: { $gte: 18, $lt: 30 } });

        // IN operator
        const admins = await User.find({ role: { $in: ['admin', 'moderator'] } });

        // Regular expression
        const usersStartingWithJ = await User.find({
            name: /^J/i  // Case insensitive
        });

        // Chain methods
        const result = await User
            .find({ isActive: true })
            .select('name email')
            .sort('-createdAt')
            .limit(10)
            .skip(0);

        return result;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

RELATIONSHIPS

// One-to-Many: User has many Posts

const postSchema = new mongoose.Schema({
    title: String,
    content: String,
    author: {
        type: mongoose.Schema.Types.ObjectId,
        ref: 'User',
        required: true
    }
}, { timestamps: true });

const Post = mongoose.model('Post', postSchema);

// Create post with user reference
async function createPost(userId) {
    const post = await Post.create({
        title: 'My First Post',
        content: 'This is the content',
        author: userId
    });
    return post;
}

// Populate (join)
async function getPostsWithAuthor() {
    const posts = await Post.find().populate('author', 'name email');
    // Only include name and email from user
    console.log(posts);
}

// Nested populate
async function getPostsWithDetails() {
    const posts = await Post.find()
        .populate({
            path: 'author',
            select: 'name email',
            match: { isActive: true }
        });
}

// Many-to-Many: Users and Roles

const roleSchema = new mongoose.Schema({
    name: String,
    permissions: [String]
});

const Role = mongoose.model('Role', roleSchema);

// User schema with roles
const userWithRolesSchema = new mongoose.Schema({
    name: String,
    email: String,
    roles: [{
        type: mongoose.Schema.Types.ObjectId,
        ref: 'Role'
    }]
});

AGGREGATION

async function aggregateUsers() {
    try {
        const result = await User.aggregate([
            // Stage 1: Match
            { $match: { isActive: true } },

            // Stage 2: Group
            {
                $group: {
                    _id: '$role',
                    count: { $sum: 1 },
                    avgAge: { $avg: '$age' },
                    maxAge: { $max: '$age' },
                    minAge: { $min: '$age' }
                }
            },

            // Stage 3: Sort
            { $sort: { count: -1 } },

            // Stage 4: Project
            {
                $project: {
                    role: '$_id',
                    count: 1,
                    avgAge: { $round: ['$avgAge', 2] },
                    _id: 0
                }
            }
        ]);

        console.log(result);
        return result;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

TRANSACTIONS

async function transferMoney(fromUserId, toUserId, amount) {
    const session = await mongoose.startSession();
    session.startTransaction();

    try {
        // Deduct from sender
        await User.findByIdAndUpdate(
            fromUserId,
            { $inc: { balance: -amount } },
            { session }
        );

        // Add to receiver
        await User.findByIdAndUpdate(
            toUserId,
            { $inc: { balance: amount } },
            { session }
        );

        // Commit transaction
        await session.commitTransaction();
        console.log('Transaction successful');
    } catch (error) {
        // Rollback on error
        await session.abortTransaction();
        console.error('Transaction failed:', error);
        throw error;
    } finally {
        session.endSession();
    }
}

INDEXES

// Create index
userSchema.index({ email: 1 });  // Ascending
userSchema.index({ name: 1, age: -1 });  // Compound index
userSchema.index({ email: 1 }, { unique: true });

// Text index for searching
userSchema.index({ name: 'text', email: 'text' });

// Search using text index
async function searchUsers(query) {
    const users = await User.find({
        $text: { $search: query }
    });
    return users;
}

PostgreSQL with Sequelize

Definition: PostgreSQL is a powerful open-source relational database. Sequelize is a promise-based ORM for Node.js.

npm install pg pg-hstore sequelize

** CONNECTION **

const { Sequelize, DataTypes } = require('sequelize');

// Connect to PostgreSQL
const sequelize = new Sequelize('database', 'username', 'password', {
    host: 'localhost',
    dialect: 'postgres',
    logging: false,  // Disable SQL logging
    pool: {
        max: 5,
        min: 0,
        acquire: 30000,
        idle: 10000
    }
});

// Test connection
async function testConnection() {
    try {
        await sequelize.authenticate();
        console.log('PostgreSQL connected');
    } catch (error) {
        console.error('Unable to connect:', error);
    }
}

DEFINE MODEL

const User = sequelize.define('User', {
    id: {
        type: DataTypes.INTEGER,
        primaryKey: true,
        autoIncrement: true
    },
    name: {
        type: DataTypes.STRING,
        allowNull: false,
        validate: {
            len: [3, 50]
        }
    },
    email: {
        type: DataTypes.STRING,
        allowNull: false,
        unique: true,
        validate: {
            isEmail: true
        }
    },
    password: {
        type: DataTypes.STRING,
        allowNull: false
    },
    age: {
        type: DataTypes.INTEGER,
        validate: {
            min: 18,
            max: 100
        }
    },
    role: {
        type: DataTypes.ENUM('user', 'admin', 'moderator'),
        defaultValue: 'user'
    },
    isActive: {
        type: DataTypes.BOOLEAN,
        defaultValue: true
    },
    avatar: DataTypes.STRING,
    lastLoginAt: DataTypes.DATE
}, {
    timestamps: true,  // createdAt, updatedAt
    tableName: 'users'
});

HOOKS

User.beforeCreate(async (user) => {
    const bcrypt = require('bcrypt');
    user.password = await bcrypt.hash(user.password, 10);
});

User.afterCreate((user) => {
    console.log('User created:', user.name);
});

INSTANCE METHODS

User.prototype.comparePassword = async function(candidatePassword) {
    const bcrypt = require('bcrypt');
    return await bcrypt.compare(candidatePassword, this.password);
};

CRUD OPERATIONS

// Sync database (create tables)
await sequelize.sync({ force: false });
// force: true drops existing tables

// CREATE
async function createUser() {
    try {
        const user = await User.create({
            name: 'John Doe',
            email: 'john@example.com',
            password: 'password123',
            age: 30
        });
        console.log('User created:', user.toJSON());
        return user;
    } catch (error) {
        console.error('Error:', error.message);
    }
}

// Bulk create
async function bulkCreate() {
    const users = await User.bulkCreate([
        { name: 'User 1', email: 'user1@example.com', password: 'pass1' },
        { name: 'User 2', email: 'user2@example.com', password: 'pass2' }
    ]);
    return users;
}

// READ
async function findUsers() {
    // Find all
    const users = await User.findAll();

    // Find with conditions
    const activeUsers = await User.findAll({
        where: {
            isActive: true
        }
    });

    // Find one
    const user = await User.findOne({
        where: {
            email: 'john@example.com'
        }
    });

    // Find by primary key
    const userById = await User.findByPk(1);

    // Count
    const count = await User.count({
        where: {
            isActive: true
        }
    });

    return users;
}

// Advanced queries
async function advancedQueries() {
    const { Op } = require('sequelize');

    // WHERE conditions
    const users = await User.findAll({
        where: {
            age: {
                [Op.gte]: 18,  // >= 18
                [Op.lt]: 30    // < 30
            }
        }
    });

    // OR condition
    const admins = await User.findAll({
        where: {
            [Op.or]: [
                { role: 'admin' },
                { role: 'moderator' }
            ]
        }
    });

    // AND condition
    const activeAdmins = await User.findAll({
        where: {
            [Op.and]: [
                { isActive: true },
                { role: 'admin' }
            ]
        }
    });

    // LIKE
    const usersStartingWithJ = await User.findAll({
        where: {
            name: {
                [Op.like]: 'J%'
            }
        }
    });

    // IN
    const specificUsers = await User.findAll({
        where: {
            id: {
                [Op.in]: [1, 2, 3]
            }
        }
    });

    // Select specific columns
    const names = await User.findAll({
        attributes: ['name', 'email']
    });

    // Exclude columns
    const usersNoPassword = await User.findAll({
        attributes: {
            exclude: ['password']
        }
    });

    // Order
    const sorted = await User.findAll({
        order: [
            ['age', 'DESC'],
            ['name', 'ASC']
        ]
    });

    // Limit and offset
    const page = await User.findAll({
        limit: 10,
        offset: 0
    });

    // Find and count
    const result = await User.findAndCountAll({
        where: { isActive: true },
        limit: 10,
        offset: 0
    });
    console.log('Total:', result.count);
    console.log('Rows:', result.rows);
}

// UPDATE
async function updateUser(id) {
    try {
        // Method 1: Update and return count
        const [count] = await User.update(
            { name: 'Updated Name' },
            { where: { id } }
        );
        console.log(`Updated ${count} user(s)`);

        // Method 2: Find and update
        const user = await User.findByPk(id);
        if (user) {
            user.name = 'New Name';
            await user.save();
        }
    } catch (error) {
        console.error('Error:', error.message);
    }
}

// DELETE
async function deleteUser(id) {
    try {
        const count = await User.destroy({
            where: { id }
        });
        console.log(`Deleted ${count} user(s)`);
    } catch (error) {
        console.error('Error:', error.message);
    }
}

RELATIONSHIPS

// One-to-Many: User has many Posts

const Post = sequelize.define('Post', {
    title: DataTypes.STRING,
    content: DataTypes.TEXT,
    userId: {
        type: DataTypes.INTEGER,
        references: {
            model: User,
            key: 'id'
        }
    }
});

// Define associations
User.hasMany(Post, { foreignKey: 'userId', as: 'posts' });
Post.belongsTo(User, { foreignKey: 'userId', as: 'author' });

// Create with association
async function createPostWithUser() {
    const user = await User.create({
        name: 'John',
        email: 'john@example.com',
        password: 'pass123',
        posts: [
            { title: 'Post 1', content: 'Content 1' },
            { title: 'Post 2', content: 'Content 2' }
        ]
    }, {
        include: [{ association: 'posts' }]
    });
}

// Query with include
async function getUserWithPosts(userId) {
    const user = await User.findByPk(userId, {
        include: [{
            model: Post,
            as: 'posts'
        }]
    });
    console.log(user.posts);
}

// Many-to-Many: Users and Roles

const Role = sequelize.define('Role', {
    name: DataTypes.STRING
});

const UserRole = sequelize.define('UserRole', {
    userId: DataTypes.INTEGER,
    roleId: DataTypes.INTEGER
});

User.belongsToMany(Role, { through: UserRole });
Role.belongsToMany(User, { through: UserRole });

*TRANSACTIONS *

async function transfer(fromUserId, toUserId, amount) {
    const t = await sequelize.transaction();

    try {
        const fromUser = await User.findByPk(fromUserId, { transaction: t });
        const toUser = await User.findByPk(toUserId, { transaction: t });

        await fromUser.update(
            { balance: fromUser.balance - amount },
            { transaction: t }
        );

        await toUser.update(
            { balance: toUser.balance + amount },
            { transaction: t }
        );

        await t.commit();
        console.log('Transaction successful');
    } catch (error) {
        await t.rollback();
        console.error('Transaction failed:', error);
    }
}

RAW QUERIES

async function rawQuery() {
    const [results, metadata] = await sequelize.query(
        'SELECT * FROM users WHERE age > :age',
        {
            replacements: { age: 25 },
            type: Sequelize.QueryTypes.SELECT
        }
    );
    return results;
}

MySQL with mysql2

npm install mysql2

const mysql = require('mysql2/promise');

// Create connection pool
const pool = mysql.createPool({
    host: 'localhost',
    user: 'root',
    password: 'password',
    database: 'myapp',
    waitForConnections: true,
    connectionLimit: 10,
    queueLimit: 0
});

// Execute query
async function getUsers() {
    try {
        const [rows, fields] = await pool.query('SELECT * FROM users');
        console.log(rows);
        return rows;
    } catch (error) {
        console.error('Error:', error);
    }
}

// Prepared statements (prevent SQL injection)
async function getUserById(id) {
    const [rows] = await pool.query(
        'SELECT * FROM users WHERE id = ?',
        [id]
    );
    return rows[0];
}

// Insert
async function createUser(name, email) {
    const [result] = await pool.query(
        'INSERT INTO users (name, email) VALUES (?, ?)',
        [name, email]
    );
    console.log('Inserted ID:', result.insertId);
    return result.insertId;
}

// Update
async function updateUser(id, name) {
    const [result] = await pool.query(
        'UPDATE users SET name = ? WHERE id = ?',
        [name, id]
    );
    console.log('Affected rows:', result.affectedRows);
}

// Delete
async function deleteUser(id) {
    const [result] = await pool.query(
        'DELETE FROM users WHERE id = ?',
        [id]
    );
    console.log('Deleted rows:', result.affectedRows);
}

// Transactions
async function transferMoney(fromId, toId, amount) {
    const connection = await pool.getConnection();

    try {
        await connection.beginTransaction();

        await connection.query(
            'UPDATE accounts SET balance = balance - ? WHERE user_id = ?',
            [amount, fromId]
        );

        await connection.query(
            'UPDATE accounts SET balance = balance + ? WHERE user_id = ?',
            [amount, toId]
        );

        await connection.commit();
        console.log('Transaction successful');
    } catch (error) {
        await connection.rollback();
        console.error('Transaction failed:', error);
        throw error;
    } finally {
        connection.release();
    }
}

9. Authentication & Authorization

JWT Authentication

Definition: JWT (JSON Web Token) is a compact, URL-safe means of representing claims to be transferred between two parties.

npm install jsonwebtoken bcrypt

USER MODEL

const jwt = require('jsonwebtoken');
const bcrypt = require('bcrypt');

const userSchema = new mongoose.Schema({
    name: String,
    email: {
        type: String,
        required: true,
        unique: true,
        lowercase: true
    },
    password: {
        type: String,
        required: true,
        minlength: 6
    },
    role: {
        type: String,
        enum: ['user', 'admin'],
        default: 'user'
    },
    refreshTokens: [String]  // Store refresh tokens
});

// Hash password before saving
userSchema.pre('save', async function(next) {
    if (!this.isModified('password')) return next();
    this.password = await bcrypt.hash(this.password, 10);
    next();
});

// Compare password method
userSchema.methods.comparePassword = async function(candidatePassword) {
    return await bcrypt.compare(candidatePassword, this.password);
};

// Generate JWT
userSchema.methods.generateAuthToken = function() {
    const token = jwt.sign(
        { id: this._id, role: this.role },
        process.env.JWT_SECRET,
        { expiresIn: '1h' }
    );
    return token;
};

// Generate refresh token
userSchema.methods.generateRefreshToken = function() {
    const refreshToken = jwt.sign(
        { id: this._id },
        process.env.JWT_REFRESH_SECRET,
        { expiresIn: '7d' }
    );
    return refreshToken;
};

const User = mongoose.model('User', userSchema);

REGISTER USER

app.post('/api/auth/register', async (req, res) => {
    try {
        const { name, email, password } = req.body;

        // Validation
        if (!name || !email || !password) {
            return res.status(400).json({
                error: 'All fields are required'
            });
        }

        // Check if user exists
        const existingUser = await User.findOne({ email });
        if (existingUser) {
            return res.status(400).json({
                error: 'Email already registered'
            });
        }

        // Create user
        const user = new User({ name, email, password });
        await user.save();

        // Generate tokens
        const token = user.generateAuthToken();
        const refreshToken = user.generateRefreshToken();

        // Save refresh token
        user.refreshTokens.push(refreshToken);
        await user.save();

        res.status(201).json({
            message: 'User registered successfully',
            token,
            refreshToken,
            user: {
                id: user._id,
                name: user.name,
                email: user.email,
                role: user.role
            }
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

LOGIN USER

app.post('/api/auth/login', async (req, res) => {
    try {
        const { email, password } = req.body;

        // Validation
        if (!email || !password) {
            return res.status(400).json({
                error: 'Email and password are required'
            });
        }

        // Find user
        const user = await User.findOne({ email });
        if (!user) {
            return res.status(401).json({
                error: 'Invalid credentials'
            });
        }

        // Check password
        const isMatch = await user.comparePassword(password);
        if (!isMatch) {
            return res.status(401).json({
                error: 'Invalid credentials'
            });
        }

        // Generate tokens
        const token = user.generateAuthToken();
        const refreshToken = user.generateRefreshToken();

        // Save refresh token
        user.refreshTokens.push(refreshToken);
        await user.save();

        res.json({
            message: 'Login successful',
            token,
            refreshToken,
            user: {
                id: user._id,
                name: user.name,
                email: user.email,
                role: user.role
            }
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

REFRESH TOKEN

app.post('/api/auth/refresh', async (req, res) => {
    try {
        const { refreshToken } = req.body;

        if (!refreshToken) {
            return res.status(400).json({
                error: 'Refresh token required'
            });
        }

        // Verify refresh token
        const decoded = jwt.verify(refreshToken, process.env.JWT_REFRESH_SECRET);

        // Find user
        const user = await User.findById(decoded.id);
        if (!user || !user.refreshTokens.includes(refreshToken)) {
            return res.status(403).json({
                error: 'Invalid refresh token'
            });
        }

        // Generate new tokens
        const newToken = user.generateAuthToken();
        const newRefreshToken = user.generateRefreshToken();

        // Replace old refresh token
        user.refreshTokens = user.refreshTokens.filter(t => t !== refreshToken);
        user.refreshTokens.push(newRefreshToken);
        await user.save();

        res.json({
            token: newToken,
            refreshToken: newRefreshToken
        });
    } catch (error) {
        if (error.name === 'TokenExpiredError') {
            return res.status(403).json({ error: 'Refresh token expired' });
        }
        res.status(500).json({ error: error.message });
    }
});

LOGOUT

app.post('/api/auth/logout', auth, async (req, res) => {
    try {
        const { refreshToken } = req.body;

        // Remove refresh token
        req.user.refreshTokens = req.user.refreshTokens.filter(
            t => t !== refreshToken
        );
        await req.user.save();

        res.json({ message: 'Logged out successfully' });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

AUTH MIDDLEWARE

const auth = async (req, res, next) => {
    try {
        // Get token from header
        const token = req.header('Authorization')?.replace('Bearer ', '');

        if (!token) {
            return res.status(401).json({
                error: 'No authentication token provided'
            });
        }

        // Verify token
        const decoded = jwt.verify(token, process.env.JWT_SECRET);

        // Find user
        const user = await User.findById(decoded.id);
        if (!user) {
            return res.status(401).json({
                error: 'User not found'
            });
        }

        // Attach user to request
        req.user = user;
        req.token = token;
        next();
    } catch (error) {
        if (error.name === 'TokenExpiredError') {
            return res.status(401).json({ error: 'Token expired' });
        }
        if (error.name === 'JsonWebTokenError') {
            return res.status(401).json({ error: 'Invalid token' });
        }
        res.status(500).json({ error: error.message });
    }
};

*PROTECTED ROUTES *

app.get('/api/profile', auth, async (req, res) => {
    res.json({
        user: {
            id: req.user._id,
            name: req.user.name,
            email: req.user.email,
            role: req.user.role
        }
    });
});

app.put('/api/profile', auth, async (req, res) => {
    try {
        const { name } = req.body;

        req.user.name = name;
        await req.user.save();

        res.json({
            message: 'Profile updated',
            user: {
                id: req.user._id,
                name: req.user.name,
                email: req.user.email
            }
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

ROLE-BASED AUTHORIZATION

const authorize = (...roles) => {
    return (req, res, next) => {
        if (!req.user) {
            return res.status(401).json({
                error: 'Authentication required'
            });
        }

        if (!roles.includes(req.user.role)) {
            return res.status(403).json({
                error: 'Access denied. Insufficient permissions.'
            });
        }

        next();
    };
};

// Admin only route
app.delete('/api/users/:id', auth, authorize('admin'), async (req, res) => {
    try {
        await User.findByIdAndDelete(req.params.id);
        res.json({ message: 'User deleted' });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

// Multiple roles
app.get('/api/admin/stats', 
    auth, 
    authorize('admin', 'moderator'), 
    async (req, res) => {
        // Get stats logic
        res.json({ stats: {} });
    }
);

Password Reset

const crypto = require('crypto');

// User schema additions
const userSchema = new mongoose.Schema({
    // ... other fields
    resetPasswordToken: String,
    resetPasswordExpire: Date
});

// Generate reset token
userSchema.methods.getResetPasswordToken = function() {
    // Generate token
    const resetToken = crypto.randomBytes(20).toString('hex');

    // Hash and set to resetPasswordToken
    this.resetPasswordToken = crypto
        .createHash('sha256')
        .update(resetToken)
        .digest('hex');

    // Set expire (10 minutes)
    this.resetPasswordExpire = Date.now() + 10 * 60 * 1000;

    return resetToken;
};

// Request password reset
app.post('/api/auth/forgot-password', async (req, res) => {
    try {
        const { email } = req.body;

        const user = await User.findOne({ email });
        if (!user) {
            return res.status(404).json({
                error: 'No user with that email'
            });
        }

        // Get reset token
        const resetToken = user.getResetPasswordToken();
        await user.save();

        // Create reset URL
        const resetUrl = `${req.protocol}://${req.get('host')}/api/auth/reset-password/${resetToken}`;

        // Send email (covered in Email section)
        const message = `You requested a password reset. Please visit: ${resetUrl}`;

        // Send email here...

        res.json({
            message: 'Reset password email sent',
            resetToken  // Remove in production
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

// Reset password
app.put('/api/auth/reset-password/:resetToken', async (req, res) => {
    try {
        // Hash the token from URL
        const resetPasswordToken = crypto
            .createHash('sha256')
            .update(req.params.resetToken)
            .digest('hex');

        // Find user
        const user = await User.findOne({
            resetPasswordToken,
            resetPasswordExpire: { $gt: Date.now() }
        });

        if (!user) {
            return res.status(400).json({
                error: 'Invalid or expired reset token'
            });
        }

        // Set new password
        user.password = req.body.password;
        user.resetPasswordToken = undefined;
        user.resetPasswordExpire = undefined;
        await user.save();

        res.json({
            message: 'Password reset successful'
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

// Change password (authenticated user)
app.put('/api/auth/change-password', auth, async (req, res) => {
    try {
        const { currentPassword, newPassword } = req.body;

        // Verify current password
        const isMatch = await req.user.comparePassword(currentPassword);
        if (!isMatch) {
            return res.status(401).json({
                error: 'Current password is incorrect'
            });
        }

        // Update password
        req.user.password = newPassword;
        await req.user.save();

        res.json({
            message: 'Password changed successfully'
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

Passport.js

Definition: Passport is authentication middleware for Node.js with support for 500+ strategies.

npm install passport passport-local passport-jwt passport-google-oauth20

const passport = require('passport');
const LocalStrategy = require('passport-local').Strategy;
const JwtStrategy = require('passport-jwt').Strategy;
const GoogleStrategy = require('passport-google-oauth20').Strategy;
const { ExtractJwt } = require('passport-jwt');

// ===== LOCAL STRATEGY =====

passport.use(new LocalStrategy({
    usernameField: 'email',
    passwordField: 'password'
}, async (email, password, done) => {
    try {
        const user = await User.findOne({ email });

        if (!user) {
            return done(null, false, { message: 'Invalid credentials' });
        }

        const isMatch = await user.comparePassword(password);

        if (!isMatch) {
            return done(null, false, { message: 'Invalid credentials' });
        }

        return done(null, user);
    } catch (error) {
        return done(error);
    }
}));

// Login route with local strategy
app.post('/api/auth/login', (req, res, next) => {
    passport.authenticate('local', (err, user, info) => {
        if (err) {
            return res.status(500).json({ error: err.message });
        }

        if (!user) {
            return res.status(401).json({ error: info.message });
        }

        const token = user.generateAuthToken();

        res.json({
            message: 'Login successful',
            token,
            user: {
                id: user._id,
                name: user.name,
                email: user.email
            }
        });
    })(req, res, next);
});

// ===== JWT STRATEGY =====

const jwtOptions = {
    jwtFromRequest: ExtractJwt.fromAuthHeaderAsBearerToken(),
    secretOrKey: process.env.JWT_SECRET
};

passport.use(new JwtStrategy(jwtOptions, async (payload, done) => {
    try {
        const user = await User.findById(payload.id);

        if (!user) {
            return done(null, false);
        }

        return done(null, user);
    } catch (error) {
        return done(error, false);
    }
}));

// Protected route with JWT
app.get('/api/profile',
    passport.authenticate('jwt', { session: false }),
    (req, res) => {
        res.json({ user: req.user });
    }
);

// ===== GOOGLE OAUTH STRATEGY =====

passport.use(new GoogleStrategy({
    clientID: process.env.GOOGLE_CLIENT_ID,
    clientSecret: process.env.GOOGLE_CLIENT_SECRET,
    callbackURL: '/api/auth/google/callback'
}, async (accessToken, refreshToken, profile, done) => {
    try {
        // Check if user exists
        let user = await User.findOne({ googleId: profile.id });

        if (!user) {
            // Create new user
            user = await User.create({
                googleId: profile.id,
                name: profile.displayName,
                email: profile.emails[0].value,
                avatar: profile.photos[0].value
            });
        }

        return done(null, user);
    } catch (error) {
        return done(error, false);
    }
}));

// Google auth routes
app.get('/api/auth/google',
    passport.authenticate('google', {
        scope: ['profile', 'email']
    })
);

app.get('/api/auth/google/callback',
    passport.authenticate('google', { session: false }),
    (req, res) => {
        const token = req.user.generateAuthToken();

        // Redirect to frontend with token
        res.redirect(`http://localhost:3000/auth/success?token=${token}`);
    }
);

// ===== SERIALIZE/DESERIALIZE (for sessions) =====

passport.serializeUser((user, done) => {
    done(null, user.id);
});

passport.deserializeUser(async (id, done) => {
    try {
        const user = await User.findById(id);
        done(null, user);
    } catch (error) {
        done(error);
    }
});

// Initialize passport
app.use(passport.initialize());

Password Reset

// ===== PASSWORD RESET FLOW =====

const crypto = require('crypto');

// User schema additions
const userSchema = new mongoose.Schema({
    // ... other fields
    resetPasswordToken: String,
    resetPasswordExpire: Date
});

// Generate reset token
userSchema.methods.getResetPasswordToken = function() {
    // Generate token
    const resetToken = crypto.randomBytes(20).toString('hex');

    // Hash and set to resetPasswordToken
    this.resetPasswordToken = crypto
        .createHash('sha256')
        .update(resetToken)
        .digest('hex');

    // Set expire (10 minutes)
    this.resetPasswordExpire = Date.now() + 10 * 60 * 1000;

    return resetToken;
};

// Request password reset
app.post('/api/auth/forgot-password', async (req, res) => {
    try {
        const { email } = req.body;

        const user = await User.findOne({ email });
        if (!user) {
            return res.status(404).json({
                error: 'No user with that email'
            });
        }

        // Get reset token
        const resetToken = user.getResetPasswordToken();
        await user.save();

        // Create reset URL
        const resetUrl = `${req.protocol}://${req.get('host')}/api/auth/reset-password/${resetToken}`;

        // Send email (covered in Email section)
        const message = `You requested a password reset. Please visit: ${resetUrl}`;

        // Send email here...

        res.json({
            message: 'Reset password email sent',
            resetToken  // Remove in production
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

// Reset password
app.put('/api/auth/reset-password/:resetToken', async (req, res) => {
    try {
        // Hash the token from URL
        const resetPasswordToken = crypto
            .createHash('sha256')
            .update(req.params.resetToken)
            .digest('hex');

        // Find user
        const user = await User.findOne({
            resetPasswordToken,
            resetPasswordExpire: { $gt: Date.now() }
        });

        if (!user) {
            return res.status(400).json({
                error: 'Invalid or expired reset token'
            });
        }

        // Set new password
        user.password = req.body.password;
        user.resetPasswordToken = undefined;
        user.resetPasswordExpire = undefined;
        await user.save();

        res.json({
            message: 'Password reset successful'
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

// Change password (authenticated user)
app.put('/api/auth/change-password', auth, async (req, res) => {
    try {
        const { currentPassword, newPassword } = req.body;

        // Verify current password
        const isMatch = await req.user.comparePassword(currentPassword);
        if (!isMatch) {
            return res.status(401).json({
                error: 'Current password is incorrect'
            });
        }

        // Update password
        req.user.password = newPassword;
        await req.user.save();

        res.json({
            message: 'Password changed successfully'
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

10. Session Management

Express Session
Definition: Sessions allow you to store user state between HTTP requests. Data is stored on the server, and a session ID is sent to the client.
Installation

npm install express-session connect-mongo

Session Configuration

const session = require('express-session');
const MongoStore = require('connect-mongo');

app.use(session({
    secret: process.env.SESSION_SECRET,  // Change in production
    resave: false,
    saveUninitialized: false,
    cookie: {
        httpOnly: true,      // Prevent XSS
        secure: false,       // Set true in production (HTTPS)
        maxAge: 1000 * 60 * 60 * 24  // 1 day
    },
    store: MongoStore.create({
        mongoUrl: process.env.MONGODB_URI,
        touchAfter: 24 * 3600  // Update session once per 24h
    })
}));

Session Usage

// Set session data
app.post('/login', async (req, res) => {
    const { email, password } = req.body;

    // Verify credentials
    const user = await User.findOne({ email });
    if (!user || !(await user.comparePassword(password))) {
        return res.status(401).json({ error: 'Invalid credentials' });
    }

    // Store user in session
    req.session.userId = user._id;
    req.session.email = user.email;
    req.session.role = user.role;

    res.json({
        message: 'Login successful',
        user: {
            id: user._id,
            email: user.email,
            role: user.role
        }
    });
});

// Access session data
app.get('/profile', (req, res) => {
    if (!req.session.userId) {
        return res.status(401).json({ error: 'Not authenticated' });
    }

    res.json({
        userId: req.session.userId,
        email: req.session.email
    });
});

// Destroy session (logout)
app.post('/logout', (req, res) => {
    req.session.destroy((err) => {
        if (err) {
            return res.status(500).json({ error: 'Logout failed' });
        }
        res.clearCookie('connect.sid');
        res.json({ message: 'Logged out successfully' });
    });
});

Session Middleware

const requireAuth = (req, res, next) => {
    if (!req.session.userId) {
        return res.status(401).json({
            error: 'Authentication required'
        });
    }
    next();
};

const requireAdmin = (req, res, next) => {
    if (req.session.role !== 'admin') {
        return res.status(403).json({
            error: 'Admin access required'
        });
    }
    next();
};

// Protected routes
app.get('/api/profile', requireAuth, (req, res) => {
    res.json({ userId: req.session.userId });
});

app.get('/api/admin', requireAuth, requireAdmin, (req, res) => {
    res.json({ message: 'Admin panel' });
});

Session Regeneration

app.post('/login', async (req, res) => {
    const user = await authenticateUser(req.body);

    // Regenerate session (security best practice)
    req.session.regenerate((err) => {
        if (err) {
            return res.status(500).json({ error: 'Session error' });
        }

        req.session.userId = user._id;
        req.session.save((err) => {
            if (err) {
                return res.status(500).json({ error: 'Session save error' });
            }
            res.json({ message: 'Login successful' });
        });
    });
});

Cookie Parser
Installation

npm install cookie-parser

Cookie Usage

const cookieParser = require('cookie-parser');

app.use(cookieParser(process.env.COOKIE_SECRET));

// Set cookies
app.get('/set-cookie', (req, res) => {
    // Simple cookie
    res.cookie('name', 'value');

    // Cookie with options
    res.cookie('userId', '12345', {
        maxAge: 900000,         // 15 minutes
        httpOnly: true,         // Not accessible via JavaScript
        secure: true,           // HTTPS only
        signed: true,           // Signed cookie
        sameSite: 'strict',     // CSRF protection
        path: '/',              // Cookie path
        domain: '.example.com'  // Cookie domain
    });

    res.send('Cookies set');
});

// Read cookies
app.get('/get-cookie', (req, res) => {
    // Unsigned cookies
    const name = req.cookies.name;

    // Signed cookies
    const userId = req.signedCookies.userId;

    res.json({
        name,
        userId
    });
});

// Clear cookies
app.get('/clear-cookie', (req, res) => {
    res.clearCookie('name');
    res.clearCookie('userId', { path: '/' });
    res.send('Cookies cleared');
});

// Cookie-based auth
app.post('/login', async (req, res) => {
    const user = await authenticateUser(req.body);

    const token = generateToken(user);

    res.cookie('token', token, {
        httpOnly: true,
        secure: process.env.NODE_ENV === 'production',
        maxAge: 24 * 60 * 60 * 1000  // 1 day
    });

    res.json({ message: 'Login successful' });
});

// Auth middleware
const cookieAuth = (req, res, next) => {
    const token = req.signedCookies.token;

    if (!token) {
        return res.status(401).json({ error: 'Not authenticated' });
    }

    try {
        const decoded = jwt.verify(token, process.env.JWT_SECRET);
        req.userId = decoded.id;
        next();
    } catch (error) {
        res.status(401).json({ error: 'Invalid token' });
    }
};

app.get('/profile', cookieAuth, (req, res) => {
    res.json({ userId: req.userId });
});

11. File Upload & Storage

Multer

Definition**: Multer is middleware for handling multipart/form-data, primarily used for file uploads.

Installation

npm install multer

Basic Upload

const multer = require('multer');
const path = require('path');
const fs = require('fs').promises;

// Memory storage (for small files)
const upload = multer({
    storage: multer.memoryStorage(),
    limits: {
        fileSize: 5 * 1024 * 1024  // 5MB
    }
});

app.post('/upload', upload.single('file'), (req, res) => {
    if (!req.file) {
        return res.status(400).json({ error: 'No file uploaded' });
    }

    res.json({
        message: 'File uploaded',
        file: {
            originalname: req.file.originalname,
            mimetype: req.file.mimetype,
            size: req.file.size,
            buffer: req.file.buffer  // File data
        }
    });
});

Disk Storage

// Configure disk storage
const storage = multer.diskStorage({
    destination: function(req, file, cb) {
        const uploadDir = 'uploads/';
        fs.mkdir(uploadDir, { recursive: true })
            .then(() => cb(null, uploadDir))
            .catch(err => cb(err));
    },
    filename: function(req, file, cb) {
        const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
        const ext = path.extname(file.originalname);
        const basename = path.basename(file.originalname, ext);
        cb(null, `${basename}-${uniqueSuffix}${ext}`);
    }
});

const diskUpload = multer({
    storage: storage,
    limits: {
        fileSize: 10 * 1024 * 1024  // 10MB
    },
    fileFilter: function(req, file, cb) {
        // Accept only images
        if (!file.mimetype.startsWith('image/')) {
            return cb(new Error('Only images allowed'), false);
        }
        cb(null, true);
    }
});

// Single file upload
app.post('/upload', diskUpload.single('image'), async (req, res) => {
    if (!req.file) {
        return res.status(400).json({ error: 'No file uploaded' });
    }

    res.json({
        message: 'File uploaded successfully',
        file: {
            filename: req.file.filename,
            originalname: req.file.originalname,
            mimetype: req.file.mimetype,
            size: req.file.size,
            path: req.file.path
        }
    });
});

// Multiple files (same field)
app.post('/upload/multiple', diskUpload.array('images', 5), (req, res) => {
    if (!req.files || req.files.length === 0) {
        return res.status(400).json({ error: 'No files uploaded' });
    }

    const files = req.files.map(file => ({
        filename: file.filename,
        originalname: file.originalname,
        size: file.size
    }));

    res.json({
        message: `${req.files.length} files uploaded`,
        files
    });
});

// Multiple fields
app.post('/upload/mixed', 
    diskUpload.fields([
        { name: 'avatar', maxCount: 1 },
        { name: 'photos', maxCount: 5 }
    ]),
    (req, res) => {
        res.json({
            avatar: req.files['avatar'],
            photos: req.files['photos']
        });
    }
);

File Validation

const imageUpload = multer({
    storage: storage,
    limits: {
        fileSize: 5 * 1024 * 1024  // 5MB
    },
    fileFilter: (req, file, cb) => {
        const allowedTypes = ['image/jpeg', 'image/png', 'image/gif'];
        if (!allowedTypes.includes(file.mimetype)) {
            return cb(new Error('Invalid file type. Only JPEG, PNG and GIF allowed.'));
        }

        const ext = path.extname(file.originalname).toLowerCase();
        if (!['.jpg', '.jpeg', '.png', '.gif'].includes(ext)) {
            return cb(new Error('Invalid file extension.'));
        }

        cb(null, true);
    }
});

// Error handling
app.post('/upload', (req, res, next) => {
    imageUpload.single('image')(req, res, (err) => {
        if (err instanceof multer.MulterError) {
            if (err.code === 'LIMIT_FILE_SIZE') {
                return res.status(400).json({
                    error: 'File too large. Maximum size is 5MB.'
                });
            }
            return res.status(400).json({ error: err.message });
        } else if (err) {
            return res.status(400).json({ error: err.message });
        }

        if (!req.file) {
            return res.status(400).json({ error: 'No file uploaded' });
        }

        res.json({
            message: 'File uploaded successfully',
            filename: req.file.filename
        });
    });
});

Image Processing with Sharp

npm install sharp

const sharp = require('sharp');

app.post('/upload/avatar', diskUpload.single('avatar'), async (req, res) => {
    try {
        if (!req.file) {
            return res.status(400).json({ error: 'No file uploaded' });
        }

        const filename = `avatar-${Date.now()}.jpg`;
        const outputPath = path.join('uploads/avatars', filename);

        await fs.mkdir('uploads/avatars', { recursive: true });

        // Resize and optimize image
        await sharp(req.file.buffer)
            .resize(200, 200, {
                fit: 'cover',
                position: 'center'
            })
            .jpeg({ quality: 90 })
            .toFile(outputPath);

        if (req.file.path) {
            await fs.unlink(req.file.path);
        }

        res.json({
            message: 'Avatar uploaded successfully',
            filename,
            path: `/avatars/${filename}`
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

Cloud Storage (AWS S3)

npm install aws-sdk multer-s3

const AWS = require('aws-sdk');
const multerS3 = require('multer-s3');

const s3 = new AWS.S3({
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: process.env.AWS_REGION
});

const s3Upload = multer({
    storage: multerS3({
        s3: s3,
        bucket: process.env.AWS_BUCKET_NAME,
        acl: 'public-read',
        metadata: (req, file, cb) => {
            cb(null, { fieldName: file.fieldname });
        },
        key: (req, file, cb) => {
            const uniqueName = `${Date.now()}-${file.originalname}`;
            cb(null, uniqueName);
        }
    }),
    limits: {
        fileSize: 5 * 1024 * 1024
    }
});

app.post('/upload/s3', s3Upload.single('file'), (req, res) => {
    if (!req.file) {
        return res.status(400).json({ error: 'No file uploaded' });
    }

    res.json({
        message: 'File uploaded to S3',
        url: req.file.location,
        key: req.file.key
    });
});

// Delete from S3
app.delete('/upload/s3/:key', async (req, res) => {
    try {
        await s3.deleteObject({
            Bucket: process.env.AWS_BUCKET_NAME,
            Key: req.params.key
        }).promise();

        res.json({ message: 'File deleted from S3' });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

File Download

app.get('/download/:filename', (req, res) => {
    const filename = req.params.filename;
    const filePath = path.join(__dirname, 'uploads', filename);

    if (!fs.existsSync(filePath)) {
        return res.status(404).json({ error: 'File not found' });
    }

    res.download(filePath, filename, (err) => {
        if (err) {
            res.status(500).json({ error: 'Download failed' });
        }
    });
});

// Serve uploaded files
app.use('/uploads', express.static('uploads'));

// Delete file
app.delete('/files/:filename', async (req, res) => {
    try {
        const filePath = path.join(__dirname, 'uploads', req.params.filename);
        await fs.unlink(filePath);
        res.json({ message: 'File deleted successfully' });
    } catch (error) {
        if (error.code === 'ENOENT') {
            return res.status(404).json({ error: 'File not found' });
        }
        res.status(500).json({ error: error.message });
    }
});

12. Error Handling

Error Handling Basics
Synchronous Error Handling

app.get('/sync-error', (req, res) => {
    throw new Error('Synchronous error!');
    // Express will catch this automatically
});

Asynchronous Error Handling

// Wrong way (error not caught)
app.get('/async-error', (req, res) => {
    setTimeout(() => {
        throw new Error('Async error!');  // Not caught!
    }, 1000);
});

// Correct way
app.get('/async-error', (req, res, next) => {
    setTimeout(() => {
        next(new Error('Async error!'));  // Pass to error handler
    }, 1000);
});

Promise Error Handling

// Wrong way
app.get('/promise-error', (req, res) => {
    User.findById(req.params.id)
        .then(user => res.json(user));
    // Error not caught!
});

// Correct way
app.get('/promise-error', (req, res, next) => {
    User.findById(req.params.id)
        .then(user => res.json(user))
        .catch(next);  // Pass error to handler
});

Async/Await Error Handling

// Manual try-catch
app.get('/users/:id', async (req, res, next) => {
    try {
        const user = await User.findById(req.params.id);
        if (!user) {
            return res.status(404).json({ error: 'User not found' });
        }
        res.json(user);
    } catch (error) {
        next(error);
    }
});

// Wrapper to avoid try-catch
const asyncHandler = (fn) => (req, res, next) => {
    Promise.resolve(fn(req, res, next)).catch(next);
};

// Usage
app.get('/users/:id', asyncHandler(async (req, res) => {
    const user = await User.findById(req.params.id);
    if (!user) {
        throw new Error('User not found');
    }
    res.json(user);
}));

Custom Error Class

class AppError extends Error {
    constructor(message, statusCode) {
        super(message);
        this.statusCode = statusCode;
        this.isOperational = true;

        Error.captureStackTrace(this, this.constructor);
    }
}

// Usage
app.get('/users/:id', asyncHandler(async (req, res) => {
    const user = await User.findById(req.params.id);
    if (!user) {
        throw new AppError('User not found', 404);
    }
    res.json(user);
}));

Error Handler Middleware

// Must have 4 parameters
app.use((err, req, res, next) => {
    console.error(err.stack);

    res.status(err.statusCode || 500).json({
        error: {
            message: err.message,
            ...(process.env.NODE_ENV === 'development' && { stack: err.stack })
        }
    });
});

Advanced Error Handler

const errorHandler = (err, req, res, next) => {
    let error = { ...err };
    error.message = err.message;

    console.error(err);

    // Mongoose bad ObjectId
    if (err.name === 'CastError') {
        const message = 'Resource not found';
        error = new AppError(message, 404);
    }

    // Mongoose duplicate key
    if (err.code === 11000) {
        const field = Object.keys(err.keyValue)[0];
        const message = `${field} already exists`;
        error = new AppError(message, 400);
    }

    // Mongoose validation error
    if (err.name === 'ValidationError') {
        const message = Object.values(err.errors)
            .map(e => e.message)
            .join(', ');
        error = new AppError(message, 400);
    }

    // JWT errors
    if (err.name === 'JsonWebTokenError') {
        error = new AppError('Invalid token', 401);
    }

    if (err.name === 'TokenExpiredError') {
        error = new AppError('Token expired', 401);
    }

    res.status(error.statusCode || 500).json({
        success: false,
        error: error.message || 'Server Error',
        ...(process.env.NODE_ENV === 'development' && {
            stack: err.stack
        })
    });
};

app.use(errorHandler);

404 Handler

// Must be after all routes
app.use((req, res, next) => {
    const error = new AppError('Route not found', 404);
    next(error);
});

Unhandled Rejection & Uncaught Exception

process.on('unhandledRejection', (err) => {
    console.log('UNHANDLED REJECTION! Shutting down...');
    console.log(err.name, err.message);
    server.close(() => {
        process.exit(1);
    });
});

process.on('uncaughtException', (err) => {
    console.log('UNCAUGHT EXCEPTION! Shutting down...');
    console.log(err.name, err.message);
    process.exit(1);
});

Graceful Shutdown

const server = app.listen(PORT);

process.on('SIGTERM', () => {
    console.log('SIGTERM received. Closing server gracefully...');
    server.close(() => {
        console.log('Server closed');
        mongoose.connection.close(false, () => {
            console.log('MongoDB connection closed');
            process.exit(0);
        });
    });
});

13. Input Validation

Using Joi

npm install joi

const Joi = require('joi');

// Define schema
const userSchema = Joi.object({
    name: Joi.string().min(3).max(50).required(),
    email: Joi.string().email().required(),
    password: Joi.string().min(6).required(),
    age: Joi.number().integer().min(18).max(100),
    role: Joi.string().valid('user', 'admin').default('user'),
    address: Joi.object({
        street: Joi.string(),
        city: Joi.string(),
        zipCode: Joi.string().pattern(/^\d{5}$/)
    }),
    hobbies: Joi.array().items(Joi.string()),
    acceptTerms: Joi.boolean().valid(true).required()
});

// Validation middleware
const validate = (schema) => {
    return (req, res, next) => {
        const { error, value } = schema.validate(req.body, { 
            abortEarly: false 
        });

        if (error) {
            const errors = error.details.map(err => err.message);
            return res.status(400).json({ errors });
        }

        req.body = value;
        next();
    };
};

// Usage
app.post('/api/users', validate(userSchema), async (req, res) => {
    const user = await User.create(req.body);
    res.json(user);
});

// Query parameters validation
const searchSchema = Joi.object({
    q: Joi.string().min(1).required(),
    page: Joi.number().integer().min(1).default(1),
    limit: Joi.number().integer().min(1).max(100).default(10)
});

app.get('/api/search', (req, res, next) => {
    const { error, value } = searchSchema.validate(req.query);
    if (error) {
        return res.status(400).json({ error: error.details[0].message });
    }
    req.query = value;
    next();
}, (req, res) => {
    // Use validated req.query
});

Using Express-Validator

npm install express-validator

const { body, query, param, validationResult } = require('express-validator');

// Validation rules
const userValidation = [
    body('name')
        .trim()
        .notEmpty().withMessage('Name is required')
        .isLength({ min: 3, max: 50 }).withMessage('Name must be 3-50 characters'),
    body('email')
        .isEmail().withMessage('Invalid email')
        .normalizeEmail(),
    body('password')
        .isLength({ min: 6 }).withMessage('Password must be at least 6 characters')
        .matches(/\d/).withMessage('Password must contain a number'),
    body('age')
        .optional()
        .isInt({ min: 18, max: 100 }).withMessage('Age must be 18-100'),
    body('role')
        .optional()
        .isIn(['user', 'admin']).withMessage('Invalid role')
];

// Error handler middleware
const handleValidationErrors = (req, res, next) => {
    const errors = validationResult(req);
    if (!errors.isEmpty()) {
        return res.status(400).json({ 
            errors: errors.array().map(err => ({
                field: err.param,
                message: err.msg
            }))
        });
    }
    next();
};

// Usage
app.post('/api/users', 
    userValidation, 
    handleValidationErrors, 
    async (req, res) => {
        const user = await User.create(req.body);
        res.json(user);
    }
);

// Custom validators
body('email').custom(async (value) => {
    const user = await User.findOne({ email: value });
    if (user) {
        throw new Error('Email already in use');
    }
    return true;
});

// Query validation
const searchValidation = [
    query('q').notEmpty().withMessage('Search term required'),
    query('page').optional().isInt({ min: 1 }),
    query('limit').optional().isInt({ min: 1, max: 100 })
];

app.get('/api/search', searchValidation, handleValidationErrors, (req, res) => {
    // Use validated query params
});

// Param validation
const idValidation = [
    param('id').isMongoId().withMessage('Invalid ID format')
];

app.get('/api/users/:id', idValidation, handleValidationErrors, (req, res) => {
    // Use validated params
});

Sanitization

const { body, sanitize } = require('express-validator');

const sanitizeUser = [
    body('name').trim().escape(),
    body('email').normalizeEmail(),
    body('bio').trim().stripLow()
];

app.post('/api/users', sanitizeUser, (req, res) => {
    // req.body is sanitized
});

14. Security Best Practices

Helmet - Security Headers

npm install helmet

const helmet = require('helmet');

app.use(helmet());

// Custom configuration
app.use(helmet({
    contentSecurityPolicy: {
        directives: {
            defaultSrc: ["'self'"],
            styleSrc: ["'self'", "'unsafe-inline'"]
        }
    },
    hsts: {
        maxAge: 31536000,
        includeSubDomains: true
    }
}));

CORS - Cross-Origin Resource Sharing

npm install cors

const cors = require('cors');

// Allow all origins (development only)
app.use(cors());

// Specific origin
app.use(cors({
    origin: 'https://yourfrontend.com',
    credentials: true,
    optionsSuccessStatus: 200
}));

// Multiple origins
const allowedOrigins = ['https://app1.com', 'https://app2.com'];
app.use(cors({
    origin: (origin, callback) => {
        if (!origin || allowedOrigins.includes(origin)) {
            callback(null, true);
        } else {
            callback(new Error('Not allowed by CORS'));
        }
    }
}));

Data Sanitization

npm install express-mongo-sanitize xss-clean hpp

const mongoSanitize = require('express-mongo-sanitize');
const xss = require('xss-clean');
const hpp = require('hpp');

// Prevent NoSQL injection
app.use(mongoSanitize());

// Prevent XSS attacks
app.use(xss());

// Prevent HTTP Parameter Pollution
app.use(hpp());

// Whitelist parameters
app.use(hpp({
    whitelist: ['price', 'rating']
}));

Environment Variables

npm install dotenv

require('dotenv').config();

// Never expose sensitive data
const config = {
    db: process.env.DB_URI,
    jwtSecret: process.env.JWT_SECRET,
    port: process.env.PORT || 3000
};

// .env file
/*
NODE_ENV=production
PORT=3000
DB_URI=mongodb://localhost:27017/myapp
JWT_SECRET=your-secret-key
JWT_EXPIRE=1h
*/

// .gitignore
/*
node_modules/
.env
*/

SQL Injection Prevention

// Use parameterized queries
const result = await pool.query(
    'SELECT * FROM users WHERE id = $1',
    [userId]
);

// Use ORM/ODM
const user = await User.findById(userId);

// Validate and sanitize input
const userId = parseInt(req.params.id);
if (isNaN(userId)) {
    return res.status(400).json({ error: 'Invalid ID' });
}

Password Security

const bcrypt = require('bcrypt');

// Hash password
const hashedPassword = await bcrypt.hash(password, 10);

// Verify password
const isValid = await bcrypt.compare(password, hashedPassword);

// Password strength validation
const passwordSchema = Joi.string()
    .min(8)
    .pattern(/^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[@$!%*?&])[A-Za-z\d@$!%*?&]/)
    .required();

HTTPS in Production

const https = require('https');
const fs = require('fs');

const options = {
    key: fs.readFileSync('privatekey.pem'),
    cert: fs.readFileSync('certificate.pem')
};

https.createServer(options, app).listen(443);

// Redirect HTTP to HTTPS
const http = require('http');
http.createServer((req, res) => {
    res.writeHead(301, { Location: `https://${req.headers.host}${req.url}` });
    res.end();
}).listen(80);

Security Checklist

// 1. Use HTTPS
// 2. Helmet for security headers
// 3. Sanitize user input
// 4. Validate all input
// 5. Use parameterized queries
// 6. Hash passwords with bcrypt
// 7. Use JWT for authentication
// 8. Implement rate limiting
// 9. Enable CORS properly
// 10. Keep dependencies updated
// 11. Use environment variables
// 12. Implement proper error handling
// 13. Log security events
// 14. Regular security audits

15. Rate Limiting & Throttling

Express Rate Limit

npm install express-rate-limit

Basic Rate Limiting

const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
    windowMs: 15 * 60 * 1000, // 15 minutes
    max: 100, // 100 requests per window
    message: 'Too many requests from this IP',
    standardHeaders: true,
    legacyHeaders: false
});

app.use('/api/', limiter);

Different Limits for Different Routes

const createAccountLimiter = rateLimit({
    windowMs: 60 * 60 * 1000, // 1 hour
    max: 5,
    message: 'Too many accounts created from this IP'
});

app.post('/api/auth/register', createAccountLimiter, registerHandler);

// Login rate limiting
const loginLimiter = rateLimit({
    windowMs: 15 * 60 * 1000,
    max: 5,
    message: 'Too many login attempts',
    skipSuccessfulRequests: true
});

app.post('/api/auth/login', loginLimiter, loginHandler);

Custom Key Generator

const limiter = rateLimit({
    windowMs: 15 * 60 * 1000,
    max: 100,
    keyGenerator: (req) => {
        return req.user?.id || req.ip;
    }
});

Custom Response

const limiter = rateLimit({
    windowMs: 15 * 60 * 1000,
    max: 100,
    handler: (req, res) => {
        res.status(429).json({
            error: 'Too many requests',
            retryAfter: Math.ceil(req.rateLimit.resetTime / 1000)
        });
    }
});

Advanced Rate Limiting

npm install rate-limiter-flexible

const { RateLimiterMemory } = require('rate-limiter-flexible');

const rateLimiter = new RateLimiterMemory({
    points: 10, // 10 requests
    duration: 1, // per 1 second
    blockDuration: 60 // Block for 60 seconds
});

const rateLimiterMiddleware = async (req, res, next) => {
    try {
        await rateLimiter.consume(req.ip);
        next();
    } catch (error) {
        res.status(429).json({
            error: 'Too many requests',
            retryAfter: error.msBeforeNext / 1000
        });
    }
};

app.use(rateLimiterMiddleware);

Per-User Rate Limiting

const userLimiter = new RateLimiterMemory({
    points: 100,
    duration: 60
});

const userRateLimit = async (req, res, next) => {
    try {
        const userId = req.user.id;
        await userLimiter.consume(userId);
        next();
    } catch (error) {
        res.status(429).json({ error: 'User rate limit exceeded' });
    }
};

app.use('/api/users', auth, userRateLimit);

Redis-Based Rate Limiting

const { RateLimiterRedis } = require('rate-limiter-flexible');
const redis = require('redis');

const redisClient = redis.createClient({
    host: 'localhost',
    port: 6379
});

const rateLimiter = new RateLimiterRedis({
    storeClient: redisClient,
    points: 10,
    duration: 1
});

const rateLimiterMiddleware = async (req, res, next) => {
    try {
        await rateLimiter.consume(req.ip);
        next();
    } catch (error) {
        res.status(429).json({ error: 'Too many requests' });
    }
};

Sliding Window Rate Limiting

const { RateLimiterMemory } = require('rate-limiter-flexible');

const opts = {
    points: 5,
    duration: 10,
    execEvenly: false,
    blockDuration: 0
};

const rateLimiter = new RateLimiterMemory(opts);

16. Caching Strategies

Redis Caching

Definition: Caching stores frequently accessed data in memory to reduce database load and improve response times.
Installation

npm install redis

Redis Connection

const redis = require('redis');

const client = redis.createClient({
    host: 'localhost',
    port: 6379
});

await client.connect();

client.on('error', (err) => {
    console.error('Redis error:', err);
});

client.on('connect', () => {
    console.log('Redis connected');
});

Simple Caching

app.get('/api/users/:id', async (req, res) => {
    const userId = req.params.id;
    const cacheKey = `user:${userId}`;

    // Check cache
    const cached = await client.get(cacheKey);
    if (cached) {
        return res.json(JSON.parse(cached));
    }

    // Fetch from database
    const user = await User.findById(userId);

    // Store in cache (expire in 1 hour)
    await client.setEx(cacheKey, 3600, JSON.stringify(user));

    res.json(user);
});

Cache Middleware

const cache = (duration) => {
    return async (req, res, next) => {
        const key = `cache:${req.originalUrl}`;

        const cached = await client.get(key);
        if (cached) {
            return res.json(JSON.parse(cached));
        }

        res.sendResponse = res.json;
        res.json = async (body) => {
            await client.setEx(key, duration, JSON.stringify(body));
            res.sendResponse(body);
        };

        next();
    };
};

// Usage
app.get('/api/posts', cache(300), async (req, res) => {
    const posts = await Post.find();
    res.json(posts);
});

Cache Invalidation

async function updateUser(userId, data) {
    const user = await User.findByIdAndUpdate(userId, data, { new: true });

    // Invalidate cache
    await client.del(`user:${userId}`);
    await client.del('cache:/api/users');

    return user;
}

// Pattern-based invalidation
async function invalidateUserCaches(userId) {
    const keys = await client.keys(`user:${userId}:*`);
    if (keys.length > 0) {
        await client.del(keys);
    }
}

Cache Patterns

// Pattern 1: Cache-Aside (Lazy Loading)
async function getUser(userId) {
    const cached = await client.get(`user:${userId}`);
    if (cached) return JSON.parse(cached);

    const user = await User.findById(userId);
    await client.setEx(`user:${userId}`, 3600, JSON.stringify(user));
    return user;
}

// Pattern 2: Write-Through
async function createUser(userData) {
    const user = await User.create(userData);
    await client.setEx(`user:${user._id}`, 3600, JSON.stringify(user));
    return user;
}

// Pattern 3: Write-Behind (Async)
async function updateUserAsync(userId, data) {
    await client.setEx(`user:${userId}`, 3600, JSON.stringify(data));

    // Update database asynchronously
    setImmediate(async () => {
        await User.findByIdAndUpdate(userId, data);
    });
}

// Pattern 4: Refresh-Ahead
async function getUserWithRefresh(userId) {
    const cached = await client.get(`user:${userId}`);
    const ttl = await client.ttl(`user:${userId}`);

    // Refresh if TTL is low
    if (ttl < 300) {
        const user = await User.findById(userId);
        await client.setEx(`user:${userId}`, 3600, JSON.stringify(user));
    }

    return cached ? JSON.parse(cached) : null;
}

Hash Caching for Objects

// Store object as hash
await client.hSet('user:1', 'name', 'John');
await client.hSet('user:1', 'email', 'john@example.com');

// Get single field
const name = await client.hGet('user:1', 'name');

// Get all fields
const user = await client.hGetAll('user:1');

// Set multiple fields
await client.hSet('user:1', {
    name: 'John',
    email: 'john@example.com',
    age: '30'
});

List Caching

// Push to list
await client.lPush('recent:users', userId);

// Get list
const recentUsers = await client.lRange('recent:users', 0, 9);

// Trim list
await client.lTrim('recent:users', 0, 99);
Set Caching
javascript// Add to set
await client.sAdd('online:users', userId);

// Check membership
const isOnline = await client.sIsMember('online:users', userId);

// Get all members
const onlineUsers = await client.sMembers('online:users');

// Remove from set
await client.sRem('online:users', userId);

Sorted Set Caching

// Add with score (timestamp)
await client.zAdd('leaderboard', {
    score: 100,
    value: 'user:1'
});

// Get top 10
const top10 = await client.zRange('leaderboard', 0, 9, { REV: true });

// Get user rank
const rank = await client.zRevRank('leaderboard', 'user:1');

17. WebSockets & Real-Time

Socket.io

Definition: WebSockets enable real-time, bidirectional communication between clients and servers.
Installation

npm install socket.io

Basic Setup

const express = require('express');
const http = require('http');
const socketIo = require('socket.io');

const app = express();
const server = http.createServer(app);
const io = socketIo(server, {
    cors: {
        origin: 'http://localhost:3000',
        methods: ['GET', 'POST']
    }
});

// Connection
io.on('connection', (socket) => {
    console.log('User connected:', socket.id);

    // Listen for events
    socket.on('message', (data) => {
        console.log('Message received:', data);

        // Emit to sender
        socket.emit('messageReceived', { message: 'Got your message' });

        // Broadcast to all except sender
        socket.broadcast.emit('newMessage', data);

        // Emit to all including sender
        io.emit('notification', { text: 'New message' });
    });

    // Disconnect
    socket.on('disconnect', () => {
        console.log('User disconnected:', socket.id);
    });
});

server.listen(3000);

Rooms

io.on('connection', (socket) => {
    // Join room
    socket.on('joinRoom', (roomId) => {
        socket.join(roomId);
        console.log(`Socket ${socket.id} joined room ${roomId}`);

        // Emit to room
        io.to(roomId).emit('userJoined', { userId: socket.id });
    });

    // Leave room
    socket.on('leaveRoom', (roomId) => {
        socket.leave(roomId);
        io.to(roomId).emit('userLeft', { userId: socket.id });
    });

    // Send to specific room
    socket.on('roomMessage', ({ roomId, message }) => {
        io.to(roomId).emit('message', message);
    });

    // Get room members
    socket.on('getRoomMembers', async (roomId) => {
        const sockets = await io.in(roomId).fetchSockets();
        socket.emit('roomMembers', sockets.map(s => s.id));
    });
});

Namespaces

const chatNamespace = io.of('/chat');
const adminNamespace = io.of('/admin');

chatNamespace.on('connection', (socket) => {
    console.log('Connected to chat namespace');

    socket.on('message', (data) => {
        chatNamespace.emit('newMessage', data);
    });
});

adminNamespace.on('connection', (socket) => {
    console.log('Connected to admin namespace');
});

Authentication

io.use((socket, next) => {
    const token = socket.handshake.auth.token;

    if (!token) {
        return next(new Error('Authentication error'));
    }

    try {
        const decoded = jwt.verify(token, process.env.JWT_SECRET);
        socket.userId = decoded.id;
        next();
    } catch (error) {
        next(new Error('Authentication error'));
    }
});

io.on('connection', (socket) => {
    console.log(`User ${socket.userId} connected`);
});

Real-Time Chat Example

const messages = [];
const users = new Map();

io.on('connection', (socket) => {
    // Send message history
    socket.emit('messageHistory', messages);

    // User joins
    socket.on('join', (username) => {
        users.set(socket.id, username);
        socket.broadcast.emit('userJoined', username);
        io.emit('userList', Array.from(users.values()));
    });

    // New message
    socket.on('sendMessage', (message) => {
        const msg = {
            id: Date.now(),
            userId: socket.id,
            username: users.get(socket.id),
            text: message,
            timestamp: new Date()
        };

        messages.push(msg);
        io.emit('newMessage', msg);
    });

    // Typing indicator
    socket.on('typing', () => {
        socket.broadcast.emit('userTyping', users.get(socket.id));
    });

    socket.on('stopTyping', () => {
        socket.broadcast.emit('userStoppedTyping', users.get(socket.id));
    });

    // Private message
    socket.on('privateMessage', ({ recipientId, message }) => {
        socket.to(recipientId).emit('privateMessage', {
            from: users.get(socket.id),
            message
        });
    });

    // Disconnect
    socket.on('disconnect', () => {
        const username = users.get(socket.id);
        users.delete(socket.id);
        io.emit('userLeft', username);
        io.emit('userList', Array.from(users.values()));
    });
});

Broadcasting

// To all clients
io.emit('event', data);

// To all except sender
socket.broadcast.emit('event', data);

// To specific room
io.to('roomId').emit('event', data);

// To multiple rooms
io.to('room1').to('room2').emit('event', data);

// To specific socket
io.to(socketId).emit('event', data);

Client Example

// Client-side (browser)
const socket = io('http://localhost:3000', {
    auth: {
        token: 'your-jwt-token'
    }
});

socket.on('connect', () => {
    console.log('Connected:', socket.id);
});

socket.emit('message', 'Hello Server');

socket.on('newMessage', (data) => {
    console.log('New message:', data);
});

socket.on('disconnect', () => {
    console.log('Disconnected');
});

18. Message Queues

Bull Queue

Definition: Message queues handle asynchronous tasks, background jobs, and distributed systems.
Installation

npm install bull

Create Queue

const Queue = require('bull');

const emailQueue = new Queue('email', {
    redis: {
        host: 'localhost',
        port: 6379
    }
});

Add Jobs

app.post('/api/users/register', async (req, res) => {
    const user = await User.create(req.body);

    // Add email job to queue
    await emailQueue.add({
        to: user.email,
        subject: 'Welcome!',
        template: 'welcome',
        data: { name: user.name }
    }, {
        attempts: 3,
        backoff: {
            type: 'exponential',
            delay: 2000
        },
        removeOnComplete: true,
        removeOnFail: false
    });

    res.json({ message: 'User registered' });
});

Process Jobs

emailQueue.process(async (job) => {
    const { to, subject, template, data } = job.data;

    console.log(`Sending email to ${to}`);

    // Update progress
    job.progress(50);

    // Send email logic here
    await sendEmail(to, subject, template, data);

    job.progress(100);

    return { sent: true, to };
});

// Multiple processors
emailQueue.process('welcome', async (job) => {
    // Process welcome emails
});

emailQueue.process('reset-password', async (job) => {
    // Process password reset emails
});

Job Events

emailQueue.on('completed', (job, result) => {
    console.log(`Job ${job.id} completed:`, result);
});

emailQueue.on('failed', (job, error) => {
    console.error(`Job ${job.id} failed:`, error);
});

emailQueue.on('progress', (job, progress) => {
    console.log(`Job ${job.id} is ${progress}% complete`);
});

emailQueue.on('stalled', (job) => {
    console.log(`Job ${job.id} stalled`);
});

emailQueue.on('active', (job) => {
    console.log(`Job ${job.id} started`);
});

Scheduled Jobs

// Delay
await emailQueue.add(
    { type: 'reminder', userId: 123 },
    { delay: 3600000 } // Send after 1 hour
);

// Repeat
await emailQueue.add(
    { type: 'daily-report' },
    { 
        repeat: { 
            cron: '0 9 * * *' // Every day at 9 AM
        } 
    }
);

// Every 5 minutes
await emailQueue.add(
    { type: 'sync' },
    { 
        repeat: { 
            every: 300000 
        } 
    }
);

Job Priority

await emailQueue.add({ type: 'urgent' }, { priority: 1 }); // High priority
await emailQueue.add({ type: 'normal' }, { priority: 5 }); // Normal priority
await emailQueue.add({ type: 'low' }, { priority: 10 }); // Low priority

Job Management

// Get job
const job = await emailQueue.getJob(jobId);

// Get job state
const state = await job.getState();

// Remove job
await job.remove();

// Retry failed job
await job.retry();

// Get job logs
const logs = await job.log('Processing...');

// Get job data
const data = job.data;
const progress = job.progress();

Queue Monitoring

app.get('/api/admin/queues/email', async (req, res) => {
    const [waiting, active, completed, failed, delayed] = await Promise.all([
        emailQueue.getWaitingCount(),
        emailQueue.getActiveCount(),
        emailQueue.getCompletedCount(),
        emailQueue.getFailedCount(),
        emailQueue.getDelayedCount()
    ]);

    res.json({ waiting, active, completed, failed, delayed });
});

// Get jobs
app.get('/api/admin/queues/email/jobs', async (req, res) => {
    const jobs = await emailQueue.getJobs(['waiting', 'active', 'failed']);
    res.json(jobs);
});

// Clean queue
app.post('/api/admin/queues/email/clean', async (req, res) => {
    await emailQueue.clean(5000, 'completed');
    await emailQueue.clean(5000, 'failed');
    res.json({ message: 'Queue cleaned' });
});

Bull Board (Queue Dashboard)

npm install bull-board

const { createBullBoard } = require('bull-board');
const { BullAdapter } = require('bull-board/bullAdapter');

const { router } = createBullBoard([
    new BullAdapter(emailQueue)
]);

app.use('/admin/queues', router);

19. Email & Notifications

Nodemailer

Installation

npm install nodemailer

Create Transporter

const nodemailer = require('nodemailer');

const transporter = nodemailer.createTransporter({
    host: 'smtp.gmail.com',
    port: 587,
    secure: false,
    auth: {
        user: process.env.EMAIL_USER,
        pass: process.env.EMAIL_PASSWORD
    }
});

// Verify connection
transporter.verify((error, success) => {
    if (error) {
        console.error('Email error:', error);
    } else {
        console.log('Email server ready');
    }
});

Send Email

async function sendEmail(options) {
    const mailOptions = {
        from: 'Your App <noreply@yourapp.com>',
        to: options.to,
        subject: options.subject,
        text: options.text,
        html: options.html
    };

    await transporter.sendMail(mailOptions);
}

// Usage
await sendEmail({
    to: 'user@example.com',
    subject: 'Welcome!',
    text: 'Welcome to our app',
    html: '<h1>Welcome to our app</h1>'
});

Welcome Email

app.post('/api/users/register', async (req, res) => {
    const user = await User.create(req.body);

    await sendEmail({
        to: user.email,
        subject: 'Welcome to Our App!',
        html: `
            <h1>Welcome ${user.name}!</h1>
            <p>Thank you for registering.</p>
            <a href="${process.env.APP_URL}/verify/${user.verificationToken}">
                Verify your email
            </a>
        `
    });

    res.json({ message: 'User registered' });
});

Password Reset Email

async function sendPasswordResetEmail(user, resetToken) {
    const resetUrl = `${process.env.FRONTEND_URL}/reset-password/${resetToken}`;

    await sendEmail({
        to: user.email,
        subject: 'Password Reset Request',
        html: `
            <h2>Password Reset</h2>
            <p>Click the link below to reset your password:</p>
            <a href="${resetUrl}">Reset Password</a>
            <p>This link expires in 10 minutes.</p>
            <p>If you didn't request this, please ignore this email.</p>
        `
    });
}

Email Templates with Handlebars

npm install handlebars

const handlebars = require('handlebars');
const fs = require('fs').promises;

async function sendTemplateEmail(to, subject, templateName, data) {
    const templatePath = `./templates/${templateName}.html`;
    const templateSource = await fs.readFile(templatePath, 'utf-8');
    const template = handlebars.compile(templateSource);
    const html = template(data);

    await sendEmail({ to, subject, html });
}

// Usage
await sendTemplateEmail(
    user.email,
    'Welcome!',
    'welcome',
    { name: user.name, appName: 'MyApp' }
);

// templates/welcome.html
/*
<!DOCTYPE html>
<html>
<head>
    <style>
        body { font-family: Arial, sans-serif; }
        .container { max-width: 600px; margin: 0 auto; }
        .button { background: #007bff; color: white; padding: 10px 20px; }
    </style>
</head>
<body>
    <div class="container">
        <h1>Welcome {{name}}!</h1>
        <p>Thank you for joining {{appName}}.</p>
        <a href="{{verifyUrl}}" class="button">Verify Email</a>
    </div>
</body>
</html>
*/

Email with Attachments

await sendEmail({
    to: 'user@example.com',
    subject: 'Invoice',
    text: 'Please find your invoice attached.',
    attachments: [
        {
            filename: 'invoice.pdf',
            path: './invoices/invoice-123.pdf'
        },
        {
            filename: 'logo.png',
            path: './images/logo.png',
            cid: 'logo'
        }
    ],
    html: `
        <img src="cid:logo" alt="Logo">
        <p>Your invoice is attached.</p>
    `
});

Email Service Providers

// SendGrid
const sgMail = require('@sendgrid/mail');
sgMail.setApiKey(process.env.SENDGRID_API_KEY);

await sgMail.send({
    to: 'user@example.com',
    from: 'noreply@yourapp.com',
    subject: 'Welcome',
    html: '<h1>Welcome!</h1>'
});

// Mailgun
const mailgun = require('mailgun-js')({
    apiKey: process.env.MAILGUN_API_KEY,
    domain: process.env.MAILGUN_DOMAIN
});

await mailgun.messages().send({
    from: 'noreply@yourapp.com',
    to: 'user@example.com',
    subject: 'Welcome',
    html: '<h1>Welcome!</h1>'
});

Email Queue Integration

const emailQueue = new Queue('email');

emailQueue.process(async (job) => {
    const { to, subject, template, data } = job.data;
    await sendTemplateEmail(to, subject, template, data);
});

app.post('/api/users/register', async (req, res) => {
    const user = await User.create(req.body);

    // Add to queue
    await emailQueue.add({
        to: user.email,
        subject: 'Welcome!',
        template: 'welcome',
        data: { name: user.name }
    });

    res.json({ message: 'User registered' });
});

20. Payment Integration

Stripe

Definition: Stripe is a payment processing platform for online businesses.
Installation

npm install stripe

Setup

const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

Create Payment Intent

app.post('/api/payment/create-intent', auth, async (req, res) => {
    try {
        const { amount, currency = 'usd' } = req.body;

        const paymentIntent = await stripe.paymentIntents.create({
            amount: amount * 100, // Amount in cents
            currency,
            metadata: {
                userId: req.user.id,
                orderId: req.body.orderId
            }
        });

        res.json({ 
            clientSecret: paymentIntent.client_secret 
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

Webhook Handler

app.post('/api/webhook/stripe', 
    express.raw({ type: 'application/json' }),
    async (req, res) => {
        const sig = req.headers['stripe-signature'];

        let event;

        try {
            event = stripe.webhooks.constructEvent(
                req.body,
                sig,
                process.env.STRIPE_WEBHOOK_SECRET
            );
        } catch (error) {
            return res.status(400).send(`Webhook Error: ${error.message}`);
        }

        // Handle event
        switch (event.type) {
            case 'payment_intent.succeeded':
                const paymentIntent = event.data.object;
                console.log('Payment succeeded:', paymentIntent.id);

                // Update order status
                await Order.findOneAndUpdate(
                    { _id: paymentIntent.metadata.orderId },
                    { status: 'paid', paymentIntentId: paymentIntent.id }
                );

                // Send confirmation email
                await sendOrderConfirmation(paymentIntent.metadata.userId);
                break;

            case 'payment_intent.payment_failed':
                console.log('Payment failed');
                // Notify user
                break;

            case 'charge.refunded':
                const charge = event.data.object;
                console.log('Charge refunded:', charge.id);
                break;
        }

        res.json({ received: true });
    }
);

Create Subscription

app.post('/api/subscription/create', auth, async (req, res) => {
    try {
        const { priceId } = req.body;

        // Create or retrieve customer
        let customer;
        if (req.user.stripeCustomerId) {
            customer = await stripe.customers.retrieve(req.user.stripeCustomerId);
        } else {
            customer = await stripe.customers.create({
                email: req.user.email,
                metadata: { userId: req.user.id }
            });

            req.user.stripeCustomerId = customer.id;
            await req.user.save();
        }

        const subscription = await stripe.subscriptions.create({
            customer: customer.id,
            items: [{ price: priceId }],
            payment_behavior: 'default_incomplete',
            expand: ['latest_invoice.payment_intent']
        });

        res.json({
            subscriptionId: subscription.id,
            clientSecret: subscription.latest_invoice.payment_intent.client_secret
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

Cancel Subscription

app.post('/api/subscription/cancel', auth, async (req, res) => {
    try {
        const { subscriptionId } = req.body;

        const subscription = await stripe.subscriptions.update(
            subscriptionId,
            { cancel_at_period_end: true }
        );

        res.json({ 
            message: 'Subscription will cancel at period end',
            cancelAt: new Date(subscription.cancel_at * 1000)
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

Create Refund

app.post('/api/payment/refund', auth, authorize('admin'), async (req, res) => {
    try {
        const { paymentIntentId, amount } = req.body;

        const refund = await stripe.refunds.create({
            payment_intent: paymentIntentId,
            amount: amount * 100
        });

        // Update order
        await Order.findOneAndUpdate(
            { paymentIntentId },
            { 
                status: 'refunded',
                refundId: refund.id
            }
        );

        res.json({ 
            message: 'Refund created',
            refundId: refund.id
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

List Customer Payments

app.get('/api/payments', auth, async (req, res) => {
    try {
        if (!req.user.stripeCustomerId) {
            return res.json({ payments: [] });
        }

        const charges = await stripe.charges.list({
            customer: req.user.stripeCustomerId,
            limit: 10
        });

        res.json({ 
            payments: charges.data.map(charge => ({
                id: charge.id,
                amount: charge.amount / 100,
                currency: charge.currency,
                status: charge.status,
                created: new Date(charge.created * 1000)
            }))
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

PayPal Integration

npm install @paypal/checkout-server-sdk

const paypal = require('@paypal/checkout-server-sdk');

function environment() {
    const clientId = process.env.PAYPAL_CLIENT_ID;
    const clientSecret = process.env.PAYPAL_CLIENT_SECRET;

    if (process.env.NODE_ENV === 'production') {
        return new paypal.core.LiveEnvironment(clientId, clientSecret);
    }
    return new paypal.core.SandboxEnvironment(clientId, clientSecret);
}

const client = new paypal.core.PayPalHttpClient(environment());

app.post('/api/payment/paypal/create', async (req, res) => {
    const request = new paypal.orders.OrdersCreateRequest();
    request.prefer('return=representation');
    request.requestBody({
        intent: 'CAPTURE',
        purchase_units: [{
            amount: {
                currency_code: 'USD',
                value: req.body.amount
            }
        }]
    });

    try {
        const order = await client.execute(request);
        res.json({ orderId: order.result.id });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

app.post('/api/payment/paypal/capture/:orderId', async (req, res) => {
    const request = new paypal.orders.OrdersCaptureRequest(req.params.orderId);

    try {
        const capture = await client.execute(request);
        res.json({ 
            status: capture.result.status,
            captureId: capture.result.purchase_units[0].payments.captures[0].id
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

21. Logging & Monitoring

Winston Logger

Definition: Logging captures application events, errors, and information for debugging and monitoring.
Installation

npm install winston morgan

Winston Setup

const winston = require('winston');

const logger = winston.createLogger({
    level: 'info',
    format: winston.format.combine(
        winston.format.timestamp(),
        winston.format.errors({ stack: true }),
        winston.format.json()
    ),
    defaultMeta: { service: 'user-service' },
    transports: [
        new winston.transports.File({ 
            filename: 'logs/error.log', 
            level: 'error' 
        }),
        new winston.transports.File({ 
            filename: 'logs/combined.log' 
        })
    ]
});

// Console logging in development
if (process.env.NODE_ENV !== 'production') {
    logger.add(new winston.transports.Console({
        format: winston.format.simple()
    }));
}

Using Logger

logger.info('Server started', { port: 3000 });
logger.error('Database connection failed', { error: error.message });
logger.warn('Rate limit exceeded', { ip: req.ip });
logger.debug('Debug information', { data: someData });

// With metadata
logger.info('User logged in', {
    userId: user.id,
    ip: req.ip,
    timestamp: new Date()
});

Custom Log Levels

const customLevels = {
    levels: {
        error: 0,
        warn: 1,
        info: 2,
        http: 3,
        debug: 4
    },
    colors: {
        error: 'red',
        warn: 'yellow',
        info: 'green',
        http: 'magenta',
        debug: 'blue'
    }
};

const logger = winston.createLogger({
    levels: customLevels.levels,
    transports: [
        new winston.transports.Console({
            format: winston.format.combine(
                winston.format.colorize(),
                winston.format.simple()
            )
        })
    ]
});

winston.addColors(customLevels.colors);

HTTP Request Logging with Morgan

const morgan = require('morgan');

// Development logging
app.use(morgan('dev'));

// Production logging with Winston
const stream = {
    write: (message) => logger.http(message.trim())
};

app.use(morgan('combined', { stream }));

// Custom Morgan format
morgan.token('user-id', (req) => req.user?.id || 'guest');

app.use(morgan(':method :url :status :user-id :response-time ms'));

Custom Logger Middleware

app.use((req, res, next) => {
    const start = Date.now();

    res.on('finish', () => {
        const duration = Date.now() - start;
        logger.http('HTTP Request', {
            method: req.method,
            url: req.url,
            status: res.statusCode,
            duration: `${duration}ms`,
            ip: req.ip,
            userAgent: req.get('user-agent'),
            userId: req.user?.id
        });
    });

    next();
});

Error Logging

app.use((err, req, res, next) => {
    logger.error('Error occurred', {
        error: err.message,
        stack: err.stack,
        url: req.url,
        method: req.method,
        ip: req.ip,
        userId: req.user?.id
    });

    res.status(err.statusCode || 500).json({
        error: err.message
    });
});

Log Rotation

npm install winston-daily-rotate-file

const DailyRotateFile = require('winston-daily-rotate-file');

const transport = new DailyRotateFile({
    filename: 'logs/application-%DATE%.log',
    datePattern: 'YYYY-MM-DD',
    maxSize: '20m',
    maxFiles: '14d'
});

logger.add(transport);

Application Performance Monitoring (APM)

npm install newrelic

// newrelic.js
exports.config = {
    app_name: ['My Application'],
    license_key: process.env.NEW_RELIC_LICENSE_KEY,
    logging: {
        level: 'info'
    }
};

// First line of your app
require('newrelic');

22. Testing Strategies

Jest Testing

Installation

npm install --save-dev jest supertest

Unit Testing

// users.service.js
class UserService {
    async createUser(userData) {
        if (!userData.email) {
            throw new Error('Email is required');
        }
        return { id: 1, ...userData };
    }

    async getUserById(id) {
        if (id < 1) {
            throw new Error('Invalid ID');
        }
        return { id, name: 'John' };
    }
}

module.exports = UserService;

// users.service.test.js
const UserService = require('./users.service');

describe('UserService', () => {
    let userService;

    beforeEach(() => {
        userService = new UserService();
    });

    describe('createUser', () => {
        it('should create a user with valid data', async () => {
            const userData = { name: 'John', email: 'john@example.com' };
            const result = await userService.createUser(userData);

            expect(result).toHaveProperty('id');
            expect(result.name).toBe('John');
            expect(result.email).toBe('john@example.com');
        });

        it('should throw error when email is missing', async () => {
            const userData = { name: 'John' };

            await expect(userService.createUser(userData))
                .rejects
                .toThrow('Email is required');
        });
    });

    describe('getUserById', () => {
        it('should return user for valid ID', async () => {
            const result = await userService.getUserById(1);
            expect(result).toHaveProperty('id', 1);
        });

        it('should throw error for invalid ID', async () => {
            await expect(userService.getUserById(0))
                .rejects
                .toThrow('Invalid ID');
        });
    });
});

Integration Testing

const request = require('supertest');
const app = require('./app');
const mongoose = require('mongoose');

describe('User API Integration Tests', () => {
    beforeAll(async () => {
        await mongoose.connect(process.env.TEST_DB_URI);
    });

    afterAll(async () => {
        await mongoose.connection.close();
    });

    beforeEach(async () => {
        await User.deleteMany({});
    });

    describe('POST /api/users', () => {
        it('should create a new user', async () => {
            const response = await request(app)
                .post('/api/users')
                .send({
                    name: 'John Doe',
                    email: 'john@example.com',
                    password: 'password123'
                });

            expect(response.status).toBe(201);
            expect(response.body).toHaveProperty('data');
            expect(response.body.data.email).toBe('john@example.com');
        });

        it('should return 400 for invalid data', async () => {
            const response = await request(app)
                .post('/api/users')
                .send({
                    name: 'John'
                });

            expect(response.status).toBe(400);
            expect(response.body).toHaveProperty('error');
        });
    });

    describe('GET /api/users/:id', () => {
        it('should return user by ID', async () => {
            const user = await User.create({
                name: 'John',
                email: 'john@example.com',
                password: 'password123'
            });

            const response = await request(app)
                .get(`/api/users/${user._id}`);

            expect(response.status).toBe(200);
            expect(response.body.data._id).toBe(user._id.toString());
        });

        it('should return 404 for non-existent user', async () => {
            const response = await request(app)
                .get('/api/users/507f1f77bcf86cd799439011');

            expect(response.status).toBe(404);
        });
    });

    describe('Authentication Tests', () => {
        it('should login user with correct credentials', async () => {
            await User.create({
                name: 'John',
                email: 'john@example.com',
                password: 'password123'
            });

            const response = await request(app)
                .post('/api/auth/login')
                .send({
                    email: 'john@example.com',
                    password: 'password123'
                });

            expect(response.status).toBe(200);
            expect(response.body).toHaveProperty('token');
        });

        it('should require authentication for protected routes', async () => {
            const response = await request(app)
                .get('/api/profile');

            expect(response.status).toBe(401);
        });

        it('should allow access with valid token', async () => {
            const user = await User.create({
                name: 'John',
                email: 'john@example.com',
                password: 'password123'
            });

            const token = user.generateAuthToken();

            const response = await request(app)
                .get('/api/profile')
                .set('Authorization', `Bearer ${token}`);

            expect(response.status).toBe(200);
        });
    });
});

Test Coverage

# Run tests with coverage
npm test -- --coverage

# package.json
{
  "scripts": {
    "test": "jest",
    "test:watch": "jest --watch",
    "test:coverage": "jest --coverage"
  },
  "jest": {
    "testEnvironment": "node",
    "coveragePathIgnorePatterns": ["/node_modules/"],
    "collectCoverageFrom": ["src/**/*.js"]
  }
}

Mocking

// Mock database
jest.mock('./models/User');
const User = require('./models/User');

describe('User Controller', () => {
    it('should find user by ID', async () => {
        User.findById.mockResolvedValue({
            _id: '123',
            name: 'John'
        });

        const result = await getUserById('123');
        expect(result.name).toBe('John');
    });
});

// Mock external API
const axios = require('axios');
jest.mock('axios');

test('fetches data from API', async () => {
    axios.get.mockResolvedValue({ data: { id: 1 } });

    const response = await fetchDataFromAPI();
    expect(response.data.id).toBe(1);
});

23. Microservices

Service Structure

project/
├── user-service/
│   ├── src/
│   ├── package.json
│   └── Dockerfile
├── product-service/
│   ├── src/
│   ├── package.json
│   └── Dockerfile
├── order-service/
│   ├── src/
│   ├── package.json
│   └── Dockerfile
└── api-gateway/
    ├── src/
    ├── package.json
    └── Dockerfile

API Gateway

npm install http-proxy-middleware

const express = require('express');
const { createProxyMiddleware } = require('http-proxy-middleware');

const app = express();

// Route requests to services
app.use('/api/users', createProxyMiddleware({
    target: 'http://user-service:3001',
    changeOrigin: true,
    pathRewrite: {
        '^/api/users': ''
    }
}));

app.use('/api/products', createProxyMiddleware({
    target: 'http://product-service:3002',
    changeOrigin: true,
    pathRewrite: {
        '^/api/products': ''
    }
}));

app.use('/api/orders', createProxyMiddleware({
    target: 'http://order-service:3003',
    changeOrigin: true,
    pathRewrite: {
        '^/api/orders': ''
    }
}));

app.listen(3000);

Service Communication

const axios = require('axios');

// User Service
app.get('/api/users/:id/orders', async (req, res) => {
    try {
        const user = await User.findById(req.params.id);

        // Call order service
        const response = await axios.get(
            `http://order-service:3003/api/orders?userId=${user._id}`,
            {
                timeout: 5000,
                headers: {
                    'X-Request-ID': req.headers['x-request-id']
                }
            }
        );

        res.json({
            user,
            orders: response.data
        });
    } catch (error) {
        res.status(500).json({ error: error.message });
    }
});

Service Discovery

npm install consul

const consul = require('consul')();

// Register service
async function registerService() {
    await consul.agent.service.register({
        id: 'user-service-1',
        name: 'user-service',
        address: 'localhost',
        port: 3001,
        check: {
            http: 'http://localhost:3001/health',
            interval: '10s'
        }
    });
}

// Discover service
async function getServiceAddress(serviceName) {
    const result = await consul.health.service(serviceName);
    const service = result[0];
    return {
        address: service.Service.Address,
        port: service.Service.Port
    };
}

Message Broker (RabbitMQ)

npm install amqplib

const amqp = require('amqplib');

// Publisher
async function publishMessage(queue, message) {
    const connection = await amqp.connect('amqp://localhost');
    const channel = await connection.createChannel();

    await channel.assertQueue(queue, { durable: true });
    channel.sendToQueue(queue, Buffer.from(JSON.stringify(message)), {
        persistent: true
    });

    setTimeout(() => {
        connection.close();
    }, 500);
}

// Consumer
async function consumeMessages(queue, callback) {
    const connection = await amqp.connect('amqp://localhost');
    const channel = await connection.createChannel();

    await channel.assertQueue(queue, { durable: true });
    channel.prefetch(1);

    channel.consume(queue, async (msg) => {
        const content = JSON.parse(msg.content.toString());
        await callback(content);
        channel.ack(msg);
    });
}

// Usage
await publishMessage('user.created', { userId: 123, email: 'user@example.com' });

await consumeMessages('user.created', async (data) => {
    console.log('New user:', data);
    await sendWelcomeEmail(data.email);
});

24. GraphQL APIs

Installation

npm install apollo-server-express graphql

Basic Setup

const { ApolloServer, gql } = require('apollo-server-express');

// Type definitions
const typeDefs = gql`
    type User {
        id: ID!
        name: String!
        email: String!
        posts: [Post!]!
        createdAt: String!
    }

    type Post {
        id: ID!
        title: String!
        content: String!
        author: User!
        published: Boolean!
        createdAt: String!
    }

    type Query {
        users: [User!]!
        user(id: ID!): User
        posts: [Post!]!
        post(id: ID!): Post
    }

    type Mutation {
        createUser(name: String!, email: String!, password: String!): User!
        updateUser(id: ID!, name: String, email: String): User!
        deleteUser(id: ID!): Boolean!

        createPost(title: String!, content: String!, authorId: ID!): Post!
        updatePost(id: ID!, title: String, content: String): Post!
        deletePost(id: ID!): Boolean!
        publishPost(id: ID!): Post!
    }

    type Subscription {
        postCreated: Post!
        userCreated: User!
    }
`;

// Resolvers
const resolvers = {
    Query: {
        users: async () => await User.find(),
        user: async (_, { id }) => await User.findById(id),
        posts: async () => await Post.find(),
        post: async (_, { id }) => await Post.findById(id)
    },

    Mutation: {
        createUser: async (_, { name, email, password }) => {
            const user = await User.create({ name, email, password });
            pubsub.publish('USER_CREATED', { userCreated: user });
            return user;
        },

        updateUser: async (_, { id, name, email }) => {
            return await User.findByIdAndUpdate(
                id,
                { name, email },
                { new: true }
            );
        },

        deleteUser: async (_, { id }) => {
            await User.findByIdAndDelete(id);
            return true;
        },

        createPost: async (_, { title, content, authorId }) => {
            const post = await Post.create({ 
                title, 
                content, 
                author: authorId 
            });
            pubsub.publish('POST_CREATED', { postCreated: post });
            return post;
        },

        publishPost: async (_, { id }) => {
            return await Post.findByIdAndUpdate(
                id,
                { published: true },
                { new: true }
            );
        }
    },

    User: {
        posts: async (user) => await Post.find({ author: user.id })
    },

    Post: {
        author: async (post) => await User.findById(post.author)
    },

    Subscription: {
        postCreated: {
            subscribe: () => pubsub.asyncIterator(['POST_CREATED'])
        },
        userCreated: {
            subscribe: () => pubsub.asyncIterator(['USER_CREATED'])
        }
    }
};

// Create Apollo Server
const server = new ApolloServer({
    typeDefs,
    resolvers,
    context: ({ req }) => ({
        user: req.user
    })
});

await server.start();
server.applyMiddleware({ app });

Authentication

const server = new ApolloServer({
    typeDefs,
    resolvers,
    context: ({ req }) => {
        const token = req.headers.authorization || '';

        try {
            const user = jwt.verify(token.replace('Bearer ', ''), process.env.JWT_SECRET);
            return { user };
        } catch (error) {
            return {};
        }
    }
});

// Protected resolver
const resolvers = {
    Mutation: {
        deleteUser: async (_, { id }, { user }) => {
            if (!user) {
                throw new Error('Not authenticated');
            }

            if (user.role !== 'admin' && user.id !== id) {
                throw new Error('Not authorized');
            }

            await User.findByIdAndDelete(id);
            return true;
        }
    }
};

25. Docker & Containers

Dockerfile

# Dockerfile
FROM node:18-alpine

# Set working directory
WORKDIR /app

# Copy package files
COPY package*.json ./

# Install dependencies
RUN npm ci --only=production

# Copy application code
COPY . .

# Expose port
EXPOSE 3000

# Set environment
ENV NODE_ENV=production

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD node healthcheck.js

# Start application
CMD ["node", "server.js"]

.dockerignore

node_modules
npm-debug.log
.env
.git
.gitignore
README.md
.vscode
.idea
*.log
dist
coverage
.env.local
.env.test

Docker Compose

# docker-compose.yml
version: '3.8'

services:
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - MONGODB_URI=mongodb://mongo:27017/myapp
      - REDIS_URL=redis://redis:6379
    depends_on:
      - mongo
      - redis
    volumes:
      - ./uploads:/app/uploads
    restart: unless-stopped
    networks:
      - app-network

  mongo:
    image: mongo:6
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db
    environment:
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=password
    restart: unless-stopped
    networks:
      - app-network

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    restart: unless-stopped
    networks:
      - app-network

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./ssl:/etc/nginx/ssl
    depends_on:
      - app
    restart: unless-stopped
    networks:
      - app-network

networks:
  app-network:
    driver: bridge

volumes:
  mongo-data:
  redis-data:

Docker Commands

# Build image
docker build -t myapp .

# Run container
docker run -p 3000:3000 myapp

# Run with environment variables
docker run -p 3000:3000 -e NODE_ENV=production myapp

# Docker Compose
docker-compose up -d
docker-compose down
docker-compose logs -f
docker-compose ps
docker-compose restart app

# Clean up
docker system prune -a
docker volume prune

Multi-Stage Build

# Build stage
FROM node:18-alpine AS builder

WORKDIR /app

COPY package*.json ./
RUN npm ci

COPY . .
RUN npm run build

# Production stage
FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --only=production

COPY --from=builder /app/dist ./dist

EXPOSE 3000

CMD ["node", "dist/server.js"]

Health Check

// healthcheck.js
const http = require('http');

const options = {
    host: 'localhost',
    port: 3000,
    path: '/health',
    timeout: 2000
};

const request = http.request(options, (res) => {
    if (res.statusCode === 200) {
        process.exit(0);
    } else {
        process.exit(1);
    }
});

request.on('error', () => {
    process.exit(1);
});

request.end();

// Health endpoint
app.get('/health', (req, res) => {
    res.status(200).json({
        status: 'healthy',
        uptime: process.uptime(),
        timestamp: new Date().toISOString()
    });
});

26. CI/CD Pipelines

GitHub Actions
Definition: CI/CD (Continuous Integration/Continuous Deployment) automates testing, building, and deploying applications.

# .github/workflows/ci.yml
name: CI/CD Pipeline

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [16.x, 18.x, 20.x]

    steps:
    - uses: actions/checkout@v3

    - name: Setup Node.js
      uses: actions/setup-node@v3
      with:
        node-version: ${{ matrix.node-version }}
        cache: 'npm'

    - name: Install dependencies
      run: npm ci

    - name: Run linter
      run: npm run lint

    - name: Run tests
      run: npm test

    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        files: ./coverage/coverage-final.json

  build:
    needs: test
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Build Docker image
      run: docker build -t myapp:${{ github.sha }} .

    - name: Push to Docker Hub
      run: |
        echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
        docker push myapp:${{ github.sha }}

  deploy:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'

    steps:
    - name: Deploy to production
      uses: appleboy/ssh-action@master
      with:
        host: ${{ secrets.HOST }}
        username: ${{ secrets.USERNAME }}
        key: ${{ secrets.SSH_KEY }}
        script: |
          cd /var/www/myapp
          git pull origin main
          npm install --production
          pm2 restart myapp

GitLab CI

# .gitlab-ci.yml
stages:
  - test
  - build
  - deploy

variables:
  NODE_VERSION: "18"

test:
  stage: test
  image: node:${NODE_VERSION}
  before_script:
    - npm ci
  script:
    - npm run lint
    - npm test
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  only:
    - main

deploy:
  stage: deploy
  script:
    - ssh user@server "cd /app && docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA && docker-compose up -d"
  only:
    - main

27. Deployment Strategies

PM2 Process Manager

# Install PM2
npm install -g pm2

# Start application
pm2 start server.js --name myapp

# Cluster mode (use all CPU cores)
pm2 start server.js -i max

# List processes
pm2 list

# Logs
pm2 logs myapp

# Restart
pm2 restart myapp

# Stop
pm2 stop myapp

# Save configuration
pm2 save

# Startup script
pm2 startup

Ecosystem Configuration

// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'myapp',
    script: './server.js',
    instances: 'max',
    exec_mode: 'cluster',
    env: {
      NODE_ENV: 'development',
      PORT: 3000
    },
    env_production: {
      NODE_ENV: 'production',
      PORT: 8080
    },
    error_file: './logs/err.log',
    out_file: './logs/out.log',
    log_date_format: 'YYYY-MM-DD HH:mm Z',
    max_memory_restart: '1G'
  }]
};

Nginx Configuration

upstream backend {
    least_conn;
    server localhost:3001;
    server localhost:3002;
    server localhost:3003;
}

server {
    listen 80;
    server_name yourdomain.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }

    location /static {
        alias /var/www/myapp/public;
        expires 1y;
    }
}

28. Performance Optimization

Database Optimization

// Indexing
userSchema.index({ email: 1 });
userSchema.index({ name: 1, age: -1 });

// Query optimization
const users = await User.find().select('name email').lean();

// Pagination
async function paginateUsers(page = 1, limit = 10) {
    const skip = (page - 1) * limit;

    const [users, total] = await Promise.all([
        User.find().skip(skip).limit(limit),
        User.countDocuments()
    ]);

    return { users, total, page, pages: Math.ceil(total / limit) };
}

Caching

const redis = require('redis');
const client = redis.createClient();

app.get('/api/posts', async (req, res) => {
    const cacheKey = 'posts:all';

    const cached = await client.get(cacheKey);
    if (cached) {
        return res.json(JSON.parse(cached));
    }

    const posts = await Post.find();
    await client.setEx(cacheKey, 300, JSON.stringify(posts));

    res.json(posts);
});

You made it this far, and that itself says a lot.

This guide was created as a focused preparation and last-minute revision for Node.js backend. If you worked through these topics, you have already built a strong foundation. From understanding the event loop to designing scalable APIs and deployments, you have touched almost every important part of modern Node.js backend development.

Remember, real confidence comes from building and experimenting. Keep creating small projects, break things, fix them, and keep improving step by step.

You don’t need to know everything perfectly. You just need to keep moving forward.

All the best for your preparation.
Keep learning and happy coding.