DEV Community: Nishanthan K

Backpropagation in Deep Learning: A Complete, Intuitive, and Practical Guide

Nishanthan K — Wed, 17 Dec 2025 03:30:00 +0000

Introduction: Why Backpropagation Matters

Backpropagation is the learning engine behind modern deep learning.
Every time a neural network improves its predictions — from recognizing faces to generating text — the improvement comes from one mechanism: adjusting weights using gradients, and those gradients are computed through backpropagation.

Why this matters:

A neural network is essentially a giant parametric function with millions or billions of weights.
Backpropagation provides an efficient way to compute how much each weight contributed to the final error.
Without backpropagation, training a deep model would be computationally impossible — you’d need to recompute partial derivatives for every weight independently, which would scale exponentially.
Backprop makes learning practical and efficient, even for very large models.
It also forms the foundation for:
- Convolutional networks
- Transformers -Diffusion models
- Reinforcement learning
- Large language models
In short, backpropagation is the backbone of modern AI.

What a Neural Network Really Computes (Forward Pass Overview)

Before understanding backpropagation, we need a clear picture of what the network does in the forward pass because backprop is simply the reverse of this computation.
At its core, a neural network performs three steps repeatedly across layers:
1. Take inputs from the previous layer
2. Apply a linear transformation using weights and biases
3. Pass the result through a non-linear activation function

Mathematically, for a single layer:

z = Wx + b
a = f(z)

Where:
- x = input vector
- W = weight matrix
- b = bias vector
- z = linear combination (pre-activation)
- f(⋅) = activation function (ReLU, Sigmoid, etc.)
- a = output (activation)

A deep network simply repeats this process layer after layer.
Each layer transforms the input into a representation that makes the final task easier — whether that’s classification, detection, language modeling, or something else.

Why this matters for backprop:
- The forward pass creates a computational graph — a chain of operations where each output depends on previous ones.
- Backprop will move in the opposite direction through this graph, calculating how changes in each weight affect the final loss.
- Knowing the exact operations in the forward pass helps us understand how the gradients flow backward.

The Core Idea Behind Backpropagation

How should each weight in the network change to reduce the final error?”

A neural network may have millions of parameters, and each parameter influences the output in a slightly different way. Backpropagation provides a systematic method to compute these influences efficiently
Here’s the core idea in simple terms:
- Calculate the loss: After the forward pass, we compare the model’s prediction with the true label and compute how “wrong” the prediction was.
- Determine how the loss changes with respect to the output of the last layer. This tells us the immediate direction in which the model should adjust.
- Move backward layer by layer
  - Using the chain rule, we figure out how the loss depends on:
    1. each layer’s activations
    2. each layer’s weights
    3. each layer’s inputs
- Accumulate gradients
  - Every weight gets a gradient value that tells us:
    - direction (positive or negative)
    - magnitude (how strong the update should be)
- Update the weights
  - Optimizers like SGD or Adam use these gradients to move the weights to slightly better values.

The key insight:

Backprop does not compute gradients from scratch for each weight.
It reuses intermediate results from the forward pass, which makes the entire process efficient.

Understanding the Computational Graph

A neural network can be viewed as a computational graph — a sequence of operations where each node represents a mathematical function, and each edge represents the flow of data.

Why this graph matters:

Every value produced during the forward pass (like z, a, activations, etc.) becomes part of the graph.
Backpropagation relies on this graph to know how outputs depend on earlier computations.
Deep learning frameworks like PyTorch and TensorFlow automatically build this graph behind the scenes.

Chain Rule: The Mathematical Engine of Backpropagation

Backpropagation is built entirely on one mathematical principle: the chain rule of calculus.
If you understand the chain rule, you understand the heart of backprop.

Why the chain rule is needed

In a neural network, the loss doesn’t depend on any weight directly.
Instead, it depends on:
- the activations of the last layer
- which depend on the previous layer
- which depend on the one before it
This creates a long chain of dependencies.
The chain rule tells us how a change in any early variable (like a weight) affects the final loss.

Applying the chain rule to layers

Consider one layer:

    z = Wx + b
    a = f(z)

And the loss:

    L = Loss(a)

Using the chain rule, the gradient of the loss with respect to the weights is:

    ∂L/∂W = (∂L/∂a) · (∂a/∂z) · (∂z/∂W)

    where
        ∂L/∂a: how the loss changes with the layer’s output
        ∂a/∂z: how the activation reacts to its input
        ∂z/∂W: how the output of this layer depends on its weights

Step-by-Step Backpropagation Through a Single Neuron

Now that the chain rule and computational graph ideas are clear, let’s walk through backpropagaion inside one neuron.
This is the smallest unit of a neural network, and understanding it removes most of the confusion around deeper models.

Forward Pass (Single Neuron)

A neuron computes:

    z = Wx + b
    a = f(z)

    where
        x -> input
        W -> weight
        b -> bias
        f -> activation function
        a -> output of the neuron

    L(a): Loss function

Backward Pass (Goal: compute gradients)

We want

Gradient with respect to the activation output

    ∂L/∂a

Gradient through the activation function

    ∂L/∂z = (∂L/∂a) · f′(z)

Gradient with respect to the weights

    ∂L/∂W = (∂L/∂z) · x

Since   z = Wx + b

        ∂z/∂W = x

So,
        ∂L/∂W = ∂L/∂z * x

Gradient with respect to the bias

        ∂z/∂b = 1
So,
        ∂L/∂b = ∂L/∂z

Gradient with respect to the input

        ∂L/∂x = (∂L/∂z) · W

Conclusion

Backpropagation is more than a mathematical procedure — it’s the core mechanism that allows neural networks to learn.
Once you understand how gradients flow through a network, you gain the ability to reason about:

Why certain architectures train well?
Why others fail?
How initialization, activations, normalization, and residual connections shape gradient behavior?
What makes transformers, CNNs, and deep RNNs stable?
How to debug training issues with confidence?

The deeper your intuition for backpropagation, the more effective you become at designing and improving models.

Neural Network — A Simple, Beginner-Friendly Overview

Nishanthan K — Mon, 01 Dec 2025 03:30:00 +0000

Neural networks are at the heart of modern AI. Whether it’s image recognition, chatbots, or recommendation systems, they all rely on these interconnected structures to learn patterns from data.

What is a Neural Network?

A neural network is made up of many interconnected nodes (neurons), each acting like a tiny decision-making unit.
Every connection between neurons carries a weight, which is a parameter the model learns during training.
These weights determine how strongly one neuron influences another, and they are gradually adjusted so the network can make better predictions over time.

How Are the Values Estimated?

When training begins, each neuron starts with random weights.
These weights are updated through backpropagation, a feedback process that adjusts them based on how wrong the model’s prediction was.
The update depends on:
- Loss value: how far the prediction was from correct
- Gradient: the direction the weight should move
- Learning rate: how big each step of adjustment should be
The goal is to reduce the loss and push the gradient closer to zero over time.
Interpreting the gradient:
- Positive gradient → decrease the weight
- Negative gradient → increase the weight

Why Neural Networks?

Traditional machine learning works well when data patterns are linear, meaning they follow straight-line relationships.
Neural networks shine because they use activation functions to introduce non-linearity, enabling them to learn complex patterns — shapes, edges, behaviors, and any relationship that isn’t simple or straight.
Popular activation functions include ReLU, Sigmoid, and Softplus, each influencing how a neuron activates.

Layers in a Neural Network

Input Layer

The network begins with an input layer containing multiple nodes.
These nodes don’t learn anything — they simply pass the raw input into the network.
They are often mistaken for neurons, but their job is mainly to feed data forward.
- They take the raw data.
- They pass those numbers into the network.
- They do not apply weights, biases, or activations.
Because of this, these nodes are often mistaken for “neurons” but they’re better understood as data carriers — a doorway into the network rather than processors.

Hidden Layers

A network may have one or several hidden layers, each filled with neurons.
These are the actual learning units, transforming the input step by step to capture patterns in the data.
- It receives input from the previous layer
- Applies weights and biases
- Passes the result through an activation function
- And sends the output deeper into the network
This repeated transformation allows the model to gradually extract patterns — edges in images, sentiment in text, or trends in numerical data.
Layer by layer, the network builds up a more meaningful internal representation of the input.

Output Layer

The final layer, responsible for producing the network’s output — a prediction, probability, class label, etc.
Depending on the problem, this layer may output:
- A single number
- A vector of probabilities
- A class label
- A multi-dimensional prediction
Like hidden neurons, output neurons also use weights, biases, and sometimes an activation function (e.g., sigmoid or softmax).
The only difference is that this layer’s output is directly interpreted as the model’s answer.

The 5-Step Life Cycle of a Neuron (Applies to Every Layer Except Input)

Every neuron inside a neural network — whether in a hidden layer or the output layer — follows the exact same processing routine. Understanding this cycle is key to building intuition.

1. It receives inputs (activations).

These activations come from either the input layer or the previous hidden layer.

2. It multiplies each input by its corresponding weight.

Weights determine how strongly the neuron should pay attention to each incoming value.

3. It adds a bias.

The bias shifts the weighted sum, giving the neuron flexibility to activate even when inputs are small or zero.

4. It applies an activation function.

The weighted sum plus bias (called the pre-activation, or z) goes through a function like ReLU, sigmoid, or tanh.
This step introduces non-linearity — essential for learning complex patterns.

5. It outputs a new activation.

This final value (a) becomes the input to the next layer, allowing the network to stack transformations and build rich representations.

Final Thoughts

A neural network is essentially a repeated cycle of: input → weighted sum → bias → activation → output layer after layer, until the final prediction is produced.

This is a gentle introduction to neural networks — enough to understand what they are made of and how they learn. From here, you can explore topics like activation functions, optimizers, loss functions, and different neural architectures.

How Neural Networks Learn – A Simple Guide to Machine Learning & Deep Learning

Nishanthan K — Fri, 28 Nov 2025 04:30:00 +0000

Machine Learning and Deep Learning are often treated as black boxes filled with complex math and jargon. But at their core, they are built on a few simple ideas: measuring error, understanding direction, and making small improvements over time.

In this article, I break down how a model actually learns — from the moment it makes a mistake, to how that mistake travels backward through the network to update weights and improve future predictions. Starting from a simple equation, we’ll build up to neural networks and the complete training loop, step by step.

1. Intelligence, Patterns, and Models

Human intelligence works by building mental models of the world.

Example:

Black clouds + strong wind → We expect rain
Sometimes this is wrong (it could be a cloud shadow)

Over time, the brain refines these models.

Machine Learning works the same way:

Input → Pattern → Output
The system learns relationships from data, not rules written manually.

A model is simply a mathematical way of representing this pattern.

2. The Core Idea: Model as an Equation

The simplest model looks like this:

y = mX + b

Where:
X = input
y = output
m and b = parameters (values to be learned)

In Machine Learning, this is often written as:

y = wX + b

Here:
w = weight (importance of input)
b = bias (base value added to the output)

The entire goal of training is to find the best values of w and b so the prediction matches reality.

3. Loss Function — Measuring Mistakes

Once the model makes a prediction, it needs to know: “How wrong am I?” That’s the job of the loss function.

A loss function:

Compares actual output and predicted output
Returns a number that shows how bad the prediction was
High loss → very wrong
Low loss → close to correct

This number becomes the main guiding signal for learning.

4. Gradient — Finding the Direction of Improvement

After loss is calculated, the model asks: “In which direction should I change my parameters to reduce this error?” The gradient is that direction.

Intuitively:

A positive gradient means: if we increase this parameter, loss will go up
A negative gradient means: if we increase this parameter, loss will go down
A zero gradient means: we are at a flat point (often a minimum)

5. Learning Rate — Controlling the Speed of Learning

The learning rate controls how much the parameter is allowed to change in each step.

Conceptually, the update rule is:

new_weight = old_weight - (learning_rate × gradient)

If the learning rate is too large → the model jumps wildly, overshoots, and becomes unstable
If the learning rate is too small → the model moves very slowly and takes forever to learn
If the learning rate is reasonable → learning is smooth and stable

So the learning rate is simply the step size of learning.

6. Epoch — Repeating the Learning Process

One epoch means the model has seen the entire training dataset once.
During an epoch, the model would makes predictions on all training data
- Calculates the loss
- Updates the parameters
- Repeats this process for the next epoch
More epochs → more opportunities to learn
Too many epochs → the model may start memorizing instead of generalizing (this is called overfitting)

7. Batch — Dividing the Dataset

Instead of feeding the entire dataset to the model at once, we split it into batches.

Example:

Dataset size: 1000 samples
Batch size: 100
Then per epoch, we have: 1000 / 100 = 10 batches

So the model updates its parameters 10 times in one epoch

Benefits of batching:

Reduces memory usage
Gives more frequent parameter updates
Often leads to better and more stable learning
There is no single “perfect” batch size. It depends on:
- Dataset size
- Available hardware (CPU/GPU, RAM/VRAM)
- Model size and complexity

8. Weights and Bias — Making Real-World Sense

Earlier, we saw:

y = mX + b

In ML terms:

y = wX + b

Where:

w (weight) represents how important an input is
b (bias) is a base value added regardless of the input

Example: Courier Charges

₹100 per kg → weight
₹50 base charge → bias
Even if the parcel weighs 0 kg, you still pay ₹50.
Without the bias term, the model would be forced to predict ₹0 when the input is 0. Bias lets us shift the line up or down and represent real “base” behavior.

So bias is needed so that:

The model is not forced to pass through (0, 0), and can better fit real-world data.

9. Neural Network — From One Equation to Many

A single neuron performs a calculation like:

y = w1·x1 + w2·x2 + ... + wn·xn + b

- It takes many inputs
- Multiplies each by its own weight
- Adds them up and adds a bias

A layer is a group of neurons working in parallel on the same input.
A neural network is a stack of such layers.

Why use neural networks?

A single linear equation is limited to straight-line relationships
Neural networks, combined with activation functions, can model complex, non-linear patterns
This is what enables them to handle images, text, audio, and other complex data

10. Activation Function — The Source of Non-Linearity

After a neuron computes:

z = w·x + b

We apply an activation function to get the final output:

a = activation(z)

Activation functions:

Introduce non-linearity
Allow the network to learn complex patterns and decision boundaries
Enable tasks such as:
- Image classification
- Object detection
- Natural language processing

Without activation functions:

Every neuron is just a linear transformation
Stacking multiple linear layers still gives a linear function
The network would be no more powerful than a single linear model

In short:

Activation functions are what make neural networks more powerful than simple linear models.

11. Forward Propagation — Data Moving Through the Network

Forward propagation is the process of passing data through the network to produce an output.
Steps:
- Input data is fed into the input layer
- Each neuron in the next layer:
- Multiplies inputs by weights
- Adds bias
- Applies activation function
The output of one layer becomes the input to the next layer
This continues until the final output layer produces the prediction
Flow: Input → Hidden layer 1 → Hidden layer 2 → ... → Output layer

Important:

Forward propagation only computes the outputs.
No learning or weight updates happen in this step.

12. Backward Propagation — Learning From Mistakes

After forward propagation, we:
- Compare the prediction with the actual target
- Compute the loss (error)
- Now the model needs to adjust its weights to reduce this loss.
- This is where backward propagation (backprop) comes in.
Backpropagation:
- Sends the error signal backward from the output layer toward the input layer
- Computes, for each weight and bias, how much it contributed to the error (its gradient)
- Once gradients are known, we update parameters using gradient descent:
  - new_weight = old_weight - (learning_rate × gradient)
  - new_bias = old_bias - (learning_rate × gradient)
- So:
  - Backpropagation calculates gradients;
  - Gradient descent + learning rate perform the actual update.
- Backpropagation is the core of how the network learns.

13. The Complete Training Cycle

Putting everything together, a full training loop for a neural network looks like this:

Input data is passed into the network
Forward propagation computes the prediction
Loss function measures how wrong the prediction
Backward propagation computes gradients for all weights and biases
Gradient descent updates the parameters using the learning rate
This is repeated for:
- Every batch
- Across multiple epochs

In one line:

Input → Forward Propagation → Loss → Backpropagation → Update Weights → Repeat

With each cycle:
- The model becomes a bit less wrong
- The loss usually goes down
- The predictions improve
- This is the entire training process in a nutshell.

14. Inference — Using the Trained Model

Once the model is trained:
- We freeze the learned weights and biases
- We give it new input data it has never seen
- It runs only forward propagation to produce predictions
- No backpropagation or weight updates happen during inference.
This is the phase where the model is actually used in production:
- Recommending products
- Classifying images
- Translating text
- Detecting fraud, etc.

Final Thought

From a simple line like y = wX + b to deep neural networks, everything in Machine Learning and Deep Learning is built on a few core ideas:

- Representing patterns with models
- Measuring error with a loss function
- Using gradients to find a better direction
- Controlling change with learning rate
- Repeating the process over batches and epochs

The concepts are simple.
The power comes from scale, data, and iteration.

Understanding Java Constructors: Types, Execution, and Differences from Methods

Nishanthan K — Sun, 05 Oct 2025 18:30:00 +0000

Understanding Constructors in Java

A constructor in Java is similar to a method but serves a specific purpose: initializing objects. This article provides a deep dive into constructors, their execution, types, and how they differ from regular methods.

What is a Constructor?

Same name as the class
No return type (not even void)
Primary role: Assign or initialize values to class variables
Automatic invocation: Called automatically when an object is created; cannot be called like a regular method

Execution of Constructors

Compile Time

The Java compiler converts a constructor into a special method called <init> in the .class file.
JVM maintains two special methods for initialization:
- <init> → Contains instance constructors; not callable directly by user code
- <clinit> → Contains static field initializations and static blocks; executed once when the class is loaded
If no constructor is defined by the developer, Java automatically creates a default constructor and stores it in <init>.

Runtime

In the JVM, a constructor is represented as a special method (<init>) and stored once in the Method Area.
Every time an object is created, the JVM executes this stored constructor; no new constructor code is created per object.
Constructors are primarily for initializing object state; heavy business logic should generally be avoided.

Types of Constructors

Default Constructor
- Added automatically by Java when no constructor is defined.
No-Argument Constructor
- Explicitly written by the developer without parameters.
Parameterized Constructor
- Written with parameters to initialize object fields flexibly.
Copy Constructor
- Not built into Java by default, but can be created by the developer to copy values from another object.

// Default constructor (auto-generated by Java)
class A {}

// No-arg constructor
class B {
    B() {
        System.out.println("B created");
    }
}

// Parameterized constructor
class C {
    String name;
    C(String n) {
        name = n;
    }
}

// Copy constructor
class D {
    String name;
    D(D other) {
        this.name = other.name;
    }
}

Constructor vs Method in Java

Feature	Constructor	Method
Name	Must be same as the class	Can have any legal identifier
Return Type	None (not even `void`)	Must declare a return type (or `void`)
Invocation	Called automatically with `new`	Must be explicitly invoked (`obj.method()`)
Purpose	Initialize object state (fresh object setup)	Define behavior of an already created object
Bytecode Representation	Stored as `<init>` in `.class` file	Stored with its given name
Inheritance Behavior	Not inherited, but calls parent constructor (`super()`)	Inherited and can be overridden
Overloading	Can be overloaded	Can be overloaded and overridden
Special Rules	Can call another constructor (`this(...)` or `super(...)`) as first statement	No such restriction

Summary

Constructor → Object birth + initialization, JVM-reserved mechanism (<init>)
Method → Object behavior, regular callable unit of code

Constructors are foundational to object-oriented programming in Java, ensuring objects are properly initialized while maintaining a clean separation of initialization logic from business logic.

Brushing Up on React Hooks

Nishanthan K — Mon, 28 Jul 2025 03:33:01 +0000

React Hooks Summary

1. `useState`

memoizedState holds the current state value.
baseState is the initial state value.
dispatch is the setter function to update state.
Triggers re-render after the current call stack is empty.

2. `useEffect`

Allows you to perform side effects (API calls, subscriptions).
Cleanup is handled by returning a function inside the effect.
Cleanup runs:
- Before the component unmounts.
- Before the effect runs again (if dependencies change).

3. `useMemo`

Caches the result of a computation.
Prevents recalculating expensive operations unless dependencies change.
Useful for:
- Filtering, sorting, parsing large data.
- Memoizing derived data for performance.
Can also be used to wrap child components to prevent unnecessary re-renders (in conjunction with React.memo).

4. `React.memo` vs `useMemo`

React.memo: Prevents re-rendering a component unless props change.
useMemo: Prevents re-running a function unless dependencies change.

5. `useCallback`

Memoizes the function definition itself.
Useful when passing callbacks to memoized child components.
Prevents unnecessary re-renders due to new function references on each render.

6. `useRef`

Mutable ref object that persists for the full lifetime of the component.
Doesn't trigger re-renders.
Can be used to access DOM nodes or hold mutable variables.

7. `useReducer`

Alternative to useState for complex state logic.
Accepts a reducer function and an initial state.
Returns [state, dispatch].
Best for managing state transitions or multiple sub-values.

8. `useLayoutEffect`

Like useEffect, but runs after DOM mutation, before paint.
Use for DOM measurements, layout adjustments.
Main use case: e.g., sidebar collapse before layout shift.

9. `useImperativeHandle`

Customizes the instance value exposed by ref.
Used with forwardRef.
Allows parent to trigger child functions imperatively.
Use case: trigger child API call on parent submit button click.

10. `useDebugValue`

Used for debugging custom hooks in React DevTools.
Display custom hook values to help debugging.

How Java Stores Data in Memory: Writing Efficient Code with Primitives and Non-Primitives

Nishanthan K — Fri, 16 May 2025 03:30:00 +0000

How Java Stores Data in Memory — Primitives vs Non-Primitives

Java is a statically typed language, which means every variable’s type must be known at compile time. This allows the JVM (Java Virtual Machine) to allocate memory efficiently and enforce type safety during execution.

Primitive Types in Java

Primitive types are the building blocks of data in Java. They have fixed memory sizes and are stored in the stack. The value of a primitive can change, but its size does not.

Type	Size	Description
`byte`	1 byte	Smallest integer (–128 to 127)
`short`	2 bytes	Short integer
`int`	4 bytes	Default integer type
`long`	8 bytes	Large-range integer
`float`	4 bytes	Single-precision float
`double`	8 bytes	Double-precision float
`char`	2 bytes	Unicode character
`boolean`	1 bit*	True or False (handled as 1 byte internally)

Note: Although boolean is conceptually 1 bit, it is usually stored as **1 byte* in memory due to alignment and practical implementation constraints.*

Non-Primitive (Reference) Types

Non-primitives, also called reference types, are more complex and stored in the heap, while their references (pointers) are kept in the stack. These types include:

Type	Description
`String`	Sequence of characters (immutable)
`Arrays`	Ordered collection of elements (of any type, including primitives)
`Class`	Custom-defined objects
`Interface`	Abstract types that other classes implement
`Enum`	A fixed set of constants (internally class-like)
`Wrapper`	Object representations of primitive types (e.g., `Integer`, `Double`)
`Collections`	Data structures like `List`, `Set`, `Map`, etc. (part of Java Collections Framework)

How Java Stores Data in Memory

Java abstracts memory management with automatic garbage collection, but here’s how it works under the hood:

Stack Memory

Stores method calls, local variables, and primitive types.
Fast access due to LIFO structure.
Memory is freed automatically once a method exits.

Heap Memory

Stores objects and non-primitive values.
Accessed via references stored in the stack.
Managed by Garbage Collector (GC).
Larger and slower than stack but can store dynamic and complex structures.

How Primitive Types Are Accessed

Stored directly in the stack.
Access and modification are fast and straightforward.
Since they are fixed-size, the JVM knows exactly how much memory to allocate and retrieve.

How Non-Primitives Are Accessed

When a non-primitive is created, the reference (memory address) is stored in the stack.
The actual object is stored in the heap.
Accessing the object involves following the reference from stack to heap.

Example:

String name = "John";

name (stack) → points to → "John" (heap)

If we change name = "Jack";, now name points to a new "Jack" in heap (because String is immutable).

Memory Alignment and JVM Optimization

Java is designed with the principle: “Write once, run anywhere”, which means JVM must work on both 32-bit and 64-bit CPUs.

JVM Alignment Strategy:

Regardless of declared type, JVM aligns memory into 4-byte or 8-byte chunks depending on CPU.
Operations on smaller types (byte, short) are converted to int internally.

Why is that?

byte takes 25% of a 4-byte block; short takes 50%.
CPU needs extra steps to calculate offsets and align the memory block.
int consumes the entire 4-byte block, making storage and retrieval faster.

This is why:

JVM promotes byte and short to int during calculations and requires explicit casting to store results back.

Misconception: Using `byte`/`short` Saves Memory?

Why `byte` and `short` are less efficient:

JVM treats them as int internally.
Adds extra overhead during arithmetic and access.
Introduces hidden conversions (widening and narrowing).
Slower due to memory misalignment and CPU inefficiency.

For performance and alignment, use int as default for integers.

What About Wrapper Classes?

Wrapper classes (Integer, Double, etc.) are object versions of primitive types.

Stored in heap like any object.
Used in Collections (e.g., List<Integer>, not List<int>)
Include utility methods like parseInt, compareTo, etc.

Note: Auto-boxing and unboxing happen automatically:

int a = 10;
Integer b = a; // auto-boxing
int c = b;     // unboxing

Boxing adds memory and performance overhead, so avoid unnecessary boxing in critical code paths.

Key Takeaways

Primitive types are fixed-size and fast, stored in stack.
Non-primitives are stored in heap with reference in stack.
JVM uses int as the baseline for all smaller integral operations.
Avoid using byte and short for optimization, it backfires due to hidden processing.
Wrapper classes add object overhead; prefer primitives for performance-critical paths.
Java abstracts memory handling, but understanding this helps you write more efficient code.

Understanding Angular Rendering, Re-rendering, and Change Detection for Optimal Performance

Nishanthan K — Mon, 07 Apr 2025 04:30:00 +0000

Introduction

If you’re transitioning from React to Angular or starting with Angular as first step, understanding Angular’s rendering process, change detection mechanism, and re-rendering strategy is crucial for writing efficient and high-performance applications. Unlike React, where the virtual DOM helps optimize the re-render process, Angular’s change detection mechanism checks every component to ensure the UI reflects the latest data. This article will break down the concepts of rendering, change detection, and performance optimization in Angular, answering the questions you might have along the way.

Understanding Angular Rendering and Re-rendering

How Does Angular Render and Re-render Components?

Angular, unlike React, uses a real DOM and checks for changes during change detection. Let's break down the process:

Initial Rendering: When a component is created for the first time, Angular processes the component’s template and binds the data from the component to the template.
Change Detection: Once the component is initialized, Angular listens for changes in the data. When any data-bound property changes, Angular triggers the change detection process, checking the whole component tree from the root.
Re-rendering: When data in a component changes, Angular goes through the change detection cycle again. Unlike React, which uses a virtual DOM, Angular re-renders the DOM directly.

This brings up an interesting question: If change detection checks the whole component tree, does Angular re-render everything even if only a small part of the data changes?

Do Angular Re-renders Everything or Just the Changed Part?

Angular does indeed go through all bindings in the component, even if only a single property changes. This is less efficient compared to React’s virtual DOM diffing mechanism, where only the changed elements are re-rendered.
However, Angular uses optimizations such as OnPush change detection strategy and TrackBy in *ngFor to reduce unnecessary checks. Let's dive deeper into these strategies.

Exploring Angular's Change Detection Mechanism

How Does Change Detection Work in Angular?

Change detection in Angular follows a process where Angular traverses the component tree and checks each binding to see if it has changed. If any change is detected, Angular updates the DOM.

The default strategy is CheckAlways, which checks all components in the component tree. But Angular allows you to optimize this using OnPush.

What is OnPush?

When using OnPush change detection, Angular only checks a component for changes when:

One of its Input properties changes.
An event is triggered (e.g., button click).
An observable emits a new value.

This can significantly improve performance in large applications by avoiding unnecessary checks on components that haven’t changed.

@Component({
  selector: 'app-my-component',
  changeDetection: ChangeDetectionStrategy.OnPush,
  templateUrl: './my-component.component.html',
})
export class MyComponent {
  @Input() data: any;
}

By using OnPush, you restrict the component to be checked only when a relevant change occurs, rather than checking it on every cycle.

Why Does Angular Check All Components for Changes?

You might wonder, Why does Angular check the entire component tree? Can it just check the component where the change happened?
Angular’s default change detection works by traversing the whole component tree to check for any data changes. This design ensures that all bindings are up to date, but can become inefficient, especially in large applications.
Unlike React’s virtual DOM, Angular doesn’t maintain a previous DOM snapshot, and so it doesn't have an efficient diffing mechanism that would allow it to detect changes at a granular level.

The Challenge with Functions in Templates

Angular has a two-way binding mechanism, which helps with reactivity. But one performance pitfall is calling functions directly in templates. Functions in templates are re-evaluated on every change detection cycle, which can hurt performance if not used properly.

For example, calling a function in a template like this:

{{ computeValue() }}

Could cause the function to run repeatedly, making it inefficient. To avoid this, use getters or Signals (in Angular 16.2+) to track state more efficiently.

get computedValue(): number {
  return this.data * 10;
}

This way, the value is calculated once, and Angular can optimize re-renders.

Optimizing Angular for Better Performance

To address performance concerns, let’s talk about strategies to optimize rendering and change detection in Angular.

Use OnPush Change Detection

As mentioned earlier, the OnPush strategy significantly reduces unnecessary re-renders. It tells Angular to check the component only when its inputs or observables change, or when an event is triggered.

Use trackBy with *ngFor

When using *ngFor, Angular creates new DOM elements for every item in the list. If the list is large, this can become inefficient. You can optimize the rendering of lists by using trackBy to track which items have changed, rather than recreating the entire DOM for the list every time.

<div *ngFor="let item of items; trackBy: trackById">
  {{ item.name }}
</div>

trackById(index: number, item: any): number {
  return item.id;
}

Avoid Functions in Templates

Instead of calling functions directly in the template, use getters or Signals (in Angular 16.2+) to keep state changes efficient and avoid recomputing values on every change detection cycle.

Signals (Angular 16.2+):

Signals in Angular provide a more efficient way to manage state and trigger updates. With signals, Angular tracks the state and updates the DOM only when the value of the signal changes.

isA = signal(true);
isB = signal(false);

Use ngOnChanges and Input Setters For efficient handling of input changes, Angular provides Input Setters and ngOnChanges. Instead of checking changes manually, use these lifecycle hooks to optimize reactivity.

@Input() set data(value: any) {
  this._data = value;
}

You can use this instead of ngOnChanges to efficiently handle changes in input properties.

Conclusion

Understanding Angular’s rendering, re-rendering, and change detection mechanisms is key to building efficient applications. By using techniques like OnPush change detection, trackBy in *ngFor, avoiding functions in templates, and leveraging Signals (in Angular 16.2+), you can optimize your Angular applications for better performance.

Remember, while Angular doesn’t use a virtual DOM like React, it offers a powerful change detection system that you can fine-tune for optimal performance. Implement these strategies and take control of your app’s reactivity—leading to faster, smoother, and more efficient user experiences.

How MongoDB’s ObjectID and B-Tree Search Make Your Queries Super Fast

Nishanthan K — Wed, 19 Feb 2025 03:30:00 +0000

MongoDB uses a unique identifier, called ObjectId, for each document stored in its database. An ObjectId is a 12-byte identifier that is generated in a specific format to ensure it’s unique across distributed systems.

In this article, we will break down the structure of MongoDB's ObjectId, how it is generated, and how MongoDB uses a binary tree-like structure for indexing and searching. We will also discuss how this structure helps MongoDB perform searches efficiently, even with millions of entries.

Structure of MongoDB's ObjectId

MongoDB’s ObjectId is a 12-byte identifier, and its structure is designed to ensure uniqueness while keeping it compact. The 12 bytes are divided into four main parts:

1. Timestamp (4 bytes)

The first 4 bytes represent the timestamp when the ObjectId was generated.
This timestamp is recorded in seconds since the Unix epoch (January 1, 1970).
This ensures that ObjectIds are roughly ordered by creation time.

2. Machine Id (3 bytes)

The next 3 bytes represent a unique identifier for the machine where the ObjectId was generated.
Typically, it’s derived from the machine's IP address or a hash of the machine's network interfaces.
This will be constant for its lifetime untill the hostname and other system details got changed
This ensures that even if two MongoDB instances are running on different machines, they can generate unique ObjectIds.

3. Process Id (2 bytes)

The following 2 bytes represent the process ID of the MongoDB server process.
This helps to distinguish ObjectIds generated by different processes running on the same machine.
This is only constant for that run and it will change when the server was restarted / on re-run.

4. Counter (3 bytes)

The last 3 bytes represent an incrementing counter.
The counter starts at a random value. If multiple documents / objectIds are generated at the same timestamp (same second), the counter ensures each generated ObjectId is unique by incrementing.
If only one Document / ObjectId is generated per second, the counter will reset to another random value for the next second.
Key point: This counter ensures uniqueness when multiple ObjectId values are generated in the same second by the same machine and process.

Example of an ObjectId:

507f191e810c19729de860ea

The first 4 bytes: 507f191e represent the timestamp.
The next 3 bytes: 81 represent the machine ID.
The next 2 bytes: 0c represent the process ID.
The last 3 bytes: 19729de represent the incrementing counter.

MongoDB Index Structure and Page Layout:

MongoDB uses a B-tree-like index structure, not an real tree we are using in data structures but with certain modifications for optimization. Here's a breakdown:

1. Root Node:

The root node of the index tree stores keys (usually indexed fields like ObjectId, or other indexed fields).
This root node acts as the starting point for the search process, guiding MongoDB towards the right child node.
Key Features:
- Contains sorted keys.
- Points to child nodes which continue the search deeper into the tree.

2. Child Pages:

Child nodes (or child pages) store sorted keys and pointers to their leaf nodes or further child nodes.
These nodes are like intermediates, helping narrow down the range of values that need to be searched.
Key Features:
- Contains sorted key-value pairs.
- Contains pointers to the next level (further child pages or leaf pages).
- May be created or split as the index grows.

3. Leaf Pages:

The leaf pages are the actual locations where document references (pointers to documents) are stored.
Each entry in the leaf page contains a key (indexed field value) and a pointer to the actual document (i.e., location in the data store).
These pages are where the final results are stored, which MongoDB returns in a query.
Key Features:
- Contains the actual document references.
- Sorted by the indexed field for fast lookups.

Page Structure:

In MongoDB, index pages are typically designed to be 4 KB in size. If the page contains entries like ObjectId (a 12-byte identifier), each page can store around 317 entries. This size may vary slightly based on the size of indexed fields.

Pages are fetched from the disk into memory during searches, ensuring that queries are handled efficiently.
MongoDB uses a binary search within a page to locate the exact entry, which ensures faster retrieval of documents.

How MongoDB Manages Page Growth:

As MongoDB adds more entries to an index, it needs to split nodes to ensure that the index remains balanced.

When a page reaches its capacity, MongoDB splits the page into two and promotes the middle key to the parent node.
The middle key is the key that divides the entries in half and is used to maintain a balanced index.
As the index continues to grow, this process of splitting and promoting ensures the B-tree remains balanced.
Balancing the tree through splits and promotions ensures that search operations remain efficient and that the index is optimal for query performance.

Example of How the Index Tree Works:

Consider an index on the ObjectId field in MongoDB:

Root Node: Contains keys pointing to child nodes. For example, if you have a collection of documents with ObjectId, the root node will store keys like ObjectId:123, ObjectId:124, etc., with pointers to child nodes that contain more detailed keys.
Child Pages: These child nodes will store ranges of ObjectIds and pointers to further nodes or leaf nodes, helping narrow down the search.
Leaf Pages: Finally, the leaf pages contain the actual document pointers for specific ObjectIds. For example, a leaf page might contain a key like ObjectId:123 and a pointer to the exact location of the document in the storage.

When a query is executed, MongoDB will traverse this tree structure by following the key pointers from the root node, through child nodes, and ultimately to the leaf nodes where it finds the document references. This ensures efficient searching, even with large datasets.

Key Takeaways:

MongoDB uses a modified B-tree index structure for fast lookups.
Pages are crucial for holding index entries and documents. Pages are typically 4 KB in size and contain sorted entries.
Leaf pages contain the actual document references and are where the search ultimately lands.
As the tree grows, node splits and middle key promotions ensure that the index remains balanced and efficient for query processing.

Example: From Scratch to 17 Entries

Let’s consider the process of adding entries to a MongoDB index and how the keys move up in the tree.

Step-by-Step Tree Construction:

Insert 1, 2, 3: Initially, we add three entries, and they all fit in the root node.

[1, 2, 3]

Insert 4: Adding the fourth entry causes the root to split. The middle key (2) gets promoted to a new root.

Root: [2]
Left Child: [1]
Right Child: [3, 4]

Insert 5, 6, 7: The right child now overflows, so the middle key (6) moves up.

Root: [2, 6]
Left Child: [1]
Middle Child: [3, 4, 5]
Right Child: [7]

Insert 8, 9, 10: The right child overflows again, so the middle key (9) moves up.

Root: [2, 6, 9]
Left Child: [1]
Middle Left Child: [3, 4, 5]
Middle Right Child: [7, 8]
Right Child: [10]

Insert 11, 12, 13, 14, 15, 16, 17: We continue inserting entries, and they cause further splits, with the middle keys getting promoted up.

Root: [2, 6, 9, 13]

Left Child: [1]

Middle Left Child: [3, 4, 5]

Middle Child: [7, 8]

Right Child: [10, 11]

Final Child: [12, 14, 15, 16, 17]

Final tree strucutre

                         [6, 9, 13]
                      /    |    |    \
         [1, 2, 3, 4, 5]   [7, 8]  [10, 11]  [12, 14, 15, 16, 17]

Simplified MongoDB Search Process

Here’s a simplified summary of how MongoDB searches for documents using the index:

Start with the Index:
- MongoDB first looks at the index (similar to a table of contents in a book) to figure out where to find documents.
- The index is stored in a B-tree structure, which helps MongoDB quickly find the right spot.
Navigate the Tree:
- The root node of the tree holds pointers to the next pages (child nodes).
- The tree is split into pages, each containing key-value pairs. These keys are the indexed fields (like ObjectId, or any other field you're indexing).
Find the Relevant Range:
- When a query is made, MongoDB starts at the root node and uses binary search to find the right range of keys.
- It keeps following the pointers to the next pages until it narrows down to the correct range of keys for the search.
Determine the Page to Search:
- As MongoDB navigates the tree, it looks for the page number (stored in the index) that contains the matching _id or document key range.
- Once it finds the correct page, it jumps to that page.
Search on the Page:
- The page contains a binary search of documents stored in it. MongoDB uses the pointers in the page to go straight to the document's storage location (like the exact spot where the document is stored on disk).
Fetch the Document:
- Finally, MongoDB uses the storage location found in the index (the position pointer) to go directly to the document on the disk and fetch it.

In short:

MongoDB first uses the index (a tree structure) to find the right range of documents.
It uses binary search to find the correct page.
Then, it jumps to that page and finds the document by looking up the position pointer, directly fetching it from the disk.

This process ensures that MongoDB can quickly locate and fetch documents even if the database grows large.

Real strucuture with 64 level split

                           ┌────────────────── Root Node ───────────────────┐
                           │ Keys: [512, 1024, 1536]                         │
                           │ Child Pointers: [P1, P2, P3, P4]                │
                           └────────────────────────────────────────────────┘
                                         ↓ Choose correct child based on _id

 ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
 │                         Internal Nodes (Level 2)                                                              │
 └───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
 │ P1 → Keys: [128, 256, 384]  │ P2 → Keys: [640, 768, 896]  │ P3 → Keys: [1152, 1280, 1408]  │ P4 → Keys: [1664, 1792, 1920] │
 │ Child Pointers: [P5-P8]     │ Child Pointers: [P9-P12]    │ Child Pointers: [P13-P16]     │ Child Pointers: [P17-P20]    │

                                         ↓ Choose correct child based on _id

 ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
 │                         Internal Nodes (Level 3)                                                              │
 └───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
 │ P5 → Keys: [32, 64, 96]  │ P6 → Keys: [160, 192, 224]  │ P7 → Keys: [288, 320, 352]  │ P8 → Keys: [416, 448, 480]  │
 │ P9 → Keys: [544, 576, 608]  │ P10 → Keys: [672, 704, 736]  │ P11 → Keys: [800, 832, 864]  │ P12 → Keys: [928, 960, 992]  │
 │ P13 → Keys: [1056, 1088, 1120]  │ P14 → Keys: [1184, 1216, 1248]  │ P15 → Keys: [1312, 1344, 1376]  │ P16 → Keys: [1440, 1472, 1504]  │
 │ P17 → Keys: [1568, 1600, 1632]  │ P18 → Keys: [1728, 1760, 1792]  │ P19 → Keys: [1856, 1888, 1920]  │ P20 → Keys: [1984, 2000]  │

                                         ↓ Choose correct child based on _id

 ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
 │                         Leaf Nodes (Level 4) - Actual Document Pointers                                       │
 └───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
 │ P21 → Keys: [1-64] (D1-D64)  │ P22 → Keys: [65-128] (D65-D128)  │ P23 → Keys: [129-192] (D129-D192)  │ P24 → Keys: [193-256] (D193-D256)  │
 │ P25 → Keys: [257-320] (D257-D320)  │ P26 → Keys: [321-384] (D321-D384)  │ P27 → Keys: [385-448] (D385-D448)  │ P28 → Keys: [449-512] (D449-D512)  │
 │ P29 → Keys: [513-576] (D513-D576)  │ P30 → Keys: [577-640] (D577-D640)  │ P31 → Keys: [641-704] (D641-D704)  │ P32 → Keys: [705-768] (D705-D768)  │
 │ P33 → Keys: [769-832] (D769-D832)  │ P34 → Keys: [833-896] (D833-D896)  │ P35 → Keys: [897-960] (D897-D960)  │ P36 → Keys: [961-1024] (D961-D1024)  │
 │ P37 → Keys: [1025-1088] (D1025-D1088)  │ P38 → Keys: [1089-1152] (D1089-D1152)  │ P39 → Keys: [1153-1216] (D1153-D1216)  │ P40 → Keys: [1217-1280] (D1217-D1280)  │
 │ P41 → Keys: [1281-1344] (D1281-D1344)  │ P42 → Keys: [1345-1408] (D1345-D1408)  │ P43 → Keys: [1409-1472] (D1409-D1472)  │ P44 → Keys: [1473-1536] (D1473-D1536)  │
 │ P45 → Keys: [1537-1600] (D1537-D1600)  │ P46 → Keys: [1601-1664] (D1601-D1664)  │ P47 → Keys: [1665-1728] (D1665-D1728)  │ P48 → Keys: [1729-1792] (D1729-D1792)  │
 │ P49 → Keys: [1793-1856] (D1793-D1856)  │ P50 → Keys: [1857-1920] (D1857-D1920)  │ P51 → Keys: [1921-1984] (D1921-D1984)  │ P52 → Keys: [1985-2000] (D1985-D2000)  │

Search flow for _id 1754

Root Node: _id = 1754 falls in range [1536, 2000], follow P4
Internal Node (L2): _id = 1754 falls in range [1664, 1920], follow P18
Internal Node (L3): _id = 1754 falls in range [1728, 1792], follow P48
Leaf Node (L4): _id = 1754 is found, get Document Pointer D1754
Retrieve document from storage

Searching for an ID in MongoDB (Example with 3.6 Million Entries)

Let’s say we have a collection of 3.6 million entries and you need to find an entry with a specific ObjectId. Here’s how MongoDB would perform the search.
At each level, you have 64 choices (64 child pointers).
So for large values (like the 3.7 millionth record), this structure is efficient.

import math

total_entries = 3600000

branching_factor = 64

num_searches = math.ceil(math.log(total_entries, branching_factor))
print(num_searches)

For searching the 3.6 millionth entry, it would take approximately 4 searches (or 4 levels of traversal in the tree). This is because the B-tree structure narrows down the search space significantly at each level.

Conclusion

MongoDB’s ObjectId provides a compact, efficient way to uniquely identify documents, and its indexing system uses a B-tree-like structure to allow for fast searches. As documents are inserted, keys are promoted to parent nodes, ensuring that MongoDB maintains a balanced and efficient index structure. When searching for a specific ObjectId, MongoDB can quickly narrow down the search space using binary search on the root and child pages, resulting in fast lookups even for large collections.

By combining timestamp-based generation for ObjectIds and a binary search-based indexing structure, MongoDB is able to offer high performance for both data storage and retrieval.

Understanding the Java Execution Process: From Code to Execution

Nishanthan K — Thu, 30 Jan 2025 04:00:00 +0000

Java is widely known for its platform independence and efficient execution. This article will walk you through the entire Java execution process, from writing human-readable code to running it across different platforms. We’ll cover the roles of JDK, JVM, and JRE, as well as the steps involved in compiling and executing Java programs.

1. Key Java Components

Before diving into the execution process, it’s essential to understand the three core components in Java:

JDK (Java Development Kit)
- The JDK is a full-fledged software development kit that allows you to write, compile, and execute Java programs.
- It includes the JVM (Java Virtual Machine) and JRE (Java Runtime Environment), as well as essential tools for development, such as the compiler (javac).
- JDK is used by developers to write and compile code, which is later executed by the JVM.
JVM (Java Virtual Machine)
- The JVM is the engine that runs Java bytecode. It makes Java platform-independent by abstracting the underlying hardware and operating system.
- Java programs are compiled into bytecode, which the JVM interprets and executes, allowing the same Java program to run on any machine with a JVM.
JRE (Java Runtime Environment)
- The JRE provides the necessary libraries and resources to run Java programs, including the JVM.
- It contains the core classes like String and Array, which your Java program may depend on.
- The JRE doesn’t include development tools like the compiler, making it suitable for running Java applications but not for development.

2. The Java Execution Process

Step 1: Write the Code
- You start by writing Java code, which is typically saved in .java files. This code is human-readable and follows Java syntax.
Step 2: Compile the Code
- Once the code is ready, the javac compiler is used to convert the human-readable .java code into bytecode (stored in .class files).
- Bytecode is a binary format, which is the same for all operating systems. This bytecode can then be executed on any platform that has a JVM, ensuring platform independence.
Step 3: Execute the Bytecode with JVM
- 3.1 Loading the Bytecode
- When you attempt to execute the Java program, the JVM loads the bytecode (i.e., the .class file) into memory.
- The ClassLoader is responsible for finding and loading the class based on the class name provided by the user.
- If the class cannot be found, a ClassNotFoundException is thrown.
- If the class is found, the JVM loads it into memory. Static methods, variables, and data from the class are stored in the Method Area, a special part of the JVM memory.
- 3.2 Executing the Bytecode
- Once the class is loaded, the JVM looks for the main() method (the entry point of the program) to begin execution.
- If the main() method is found, the execution process starts.

3. Execution Mechanism

There are two main approaches the JVM uses to execute bytecode: Interpreter and Just-In-Time (JIT) Compiler.

Interpreter (Slower)
- In the interpreter approach, the JVM reads and executes the bytecode line by line.
- Every time a method is invoked, the JVM re-interprets the bytecode, which can be slow since the same method may be re-executed multiple times.
JIT (Just-In-Time) Compiler (Faster)
- The JIT Compiler compiles bytecode into native machine code, which is specific to the platform and machine on which the program is running.
- It optimizes performance by using a technique called Hot Spots.
- Hot Spots are frequently used parts of the code (like methods). These are identified by the JIT compiler, and instead of interpreting them each time, the JIT compiles them into native machine code.
- The compiled machine code is cached, so when the same method is needed again, the JVM can use the cached machine code, resulting in faster execution.
Hot Spots
- The Hot Spot technique ensures that the JVM only compiles the frequently used methods, not the entire class. This results in significant performance improvements for long-running applications.
- The JVM uses machine code for execution of these hot spots instead of interpreting the bytecode every time.

4. JVM Memory Areas

The JVM allocates memory for different parts of the program during execution. Some key areas include:
- Method Area: Holds information about classes, methods, and static variables.
- Heap Area: Stores objects created during runtime.
- Stack Area: Stores local variables and method calls.
- Program Counter (PC): A register that points to the current instruction being executed.

5. Summary of the Execution Flow

Write Code: Java code is written in .java files.
Compile: The code is compiled into bytecode (.class files) by the javac compiler.
Load Bytecode: The JVM, using the ClassLoader, loads the bytecode into memory.
Find Entry Point: The JVM looks for the main() method to start execution.
Execution via Interpreter or JIT:
- Interpreter: Executes bytecode line by line (slower).
- JIT Compiler: Compiles hot spots into native machine code for faster execution (faster).

The combination of bytecode, JVM, and the JIT compiler ensures that Java is both platform-independent and efficient. The execution flow uses the Interpreter for simplicity and the JIT Compiler for performance optimization, allowing Java programs to run efficiently across various platforms.

Recap:

The JVM plays a key role in ensuring that Java code is portable and efficient. It first loads the bytecode, then executes it through either an interpreter (slower) or a JIT compiler (faster).
By using JIT and hot spots, the JVM optimizes performance while maintaining the ability to run the same bytecode on different platforms.

Mastering Node.js Routing: A Complete Guide with Internal Workings Explained

Nishanthan K — Fri, 20 Dec 2024 04:00:00 +0000

Understanding Routes in Node.js: A Beginner's Guide

Routing is a fundamental concept in Node.js, especially when building web applications. It refers to defining endpoints in your application that respond to client requests. Each route is associated with a specific HTTP method (like GET, POST, PUT, or DELETE) and a path. Let’s dive into how routing works in Node.js and explore its internal mechanics.

Setting Up Routes in Node.js

To understand routing, let’s start with a simple example using the popular Express.js framework:

const express = require('express');
const app = express();

// Define a GET route
app.get('/', (req, res) => {
  res.send('Welcome to the Home Page!');
});

// Define a POST route
app.post('/submit', (req, res) => {
  res.send('Form Submitted Successfully!');
});

// Start the server
app.listen(3000, () => {
  console.log('Server is running on http://localhost:3000');
});

Here’s what happens:

Route Definition: Each app.method(path, handler) defines a route where:
- method specifies the HTTP method (GET, POST, etc.).
- path is the URL endpoint (e.g., /submit).
- handler is the function executed when the route is accessed.
Middleware Layer: Express uses middleware to process incoming requests before reaching the route handler. Middleware can modify the request and response objects.

Internal Working of Routes in Node.js

When a request hits the server, here’s what happens under the hood:

Request Matching:
- Node.js, with Express, maintains a routing table—a collection of defined routes.
- When a request arrives, Express parses the request's HTTP method and URL.
- It compares these against the routing table to find a match.
Middleware Execution:
- Before reaching the handler, middleware functions are executed in the order they are defined.
- Middleware can handle logging, authentication, or parsing JSON data from the request body.
Route Handler Execution:
- Once a matching route is found, its handler function is executed.
- The handler has access to:
  - req (request object): Contains data like query parameters, headers, and body.
  - res (response object): Used to send back a response to the client.
Response Dispatch:
- The res object sends a response (like HTML, JSON, or plain text) back to the client.
- Once the response is sent, the connection is closed.

Advanced Routing Techniques

Express supports advanced routing capabilities, such as:

Route Parameters:

app.get('/user/:id', (req, res) => {
  res.send(`User ID: ${req.params.id}`);
});

Query Parameters:

app.get('/search', (req, res) => {
  res.send(`Search Query: ${req.query.q}`);
});

Chained Routes:

app.route('/product')
  .get((req, res) => res.send('Get Product'))
  .post((req, res) => res.send('Add Product'));

Debugging and Performance

Use tools like morgan for logging requests.
Monitor performance with middleware like compression to optimize response sizes.

Conclusion

Routing in Node.js, especially with Express, provides a powerful way to define and manage application endpoints. Internally, it uses a systematic approach to match requests, execute middleware, and respond to clients. Understanding these mechanics can help you build more efficient and maintainable applications.

Save this article for future reference and let us know your thoughts in the comments below!

Understanding Middlewares in Express.js and Their Internal Working

Nishanthan K — Wed, 18 Dec 2024 04:00:00 +0000

In Express.js, middlewares are special functions that have access to the request (req), response (res), and a third parameter called next. Unlike regular route handlers, middlewares play a critical role in controlling the flow of the application by executing external logic before the main business logic.

How Do Middlewares Work?

When an HTTP request hits an Express.js server, it flows through a series of middleware functions. Each middleware can:

Modify the request object (e.g., attach data, validate tokens).
Modify the response object (e.g., send a response early).
Pass control to the next middleware in the stack using the next() function.

If a middleware doesn't call next(), the request-response cycle terminates there, and no further logic (including route handlers) will execute.

Why Do We Use Middlewares?

Middlewares are perfect for scenarios where we need to add reusable logic before processing a request. For instance:

Authentication: Checking if the user is logged in (e.g., validating - JWT tokens).
Authorization: Ensuring users have the necessary permissions to perform certain actions (e.g., admins can delete content).
Request Validation: Verifying if all required inputs are provided.
Logging and Monitoring: Recording details of incoming requests for analytics or debugging.
Error Handling: Catching errors globally to send meaningful responses.

Defining and Using Middlewares

A middleware function looks like this:

app.use((req, res, next) => {
    // Logic here
    next(); // Pass control to the next middleware or route handler
});

req (Request): Contains information about the incoming HTTP request (e.g., headers, body, params).
res (Response): Used to send data back to the client. next(): A function that passes control to the next middleware in line.

Middleware Flow: Order of Execution

Middleware order matters! Express executes middlewares sequentially in the order they are defined.

If a middleware is defined after a route, it will not affect that route. This is why middlewares must be declared before routes in your app.js.

Example:

// Middleware to check if the user has admin privileges
app.use((req, res, next) => {
    console.log("Checking for admin role...");

    // Simulating a user object attached earlier in the pipeline
    if (req.user && req.user.role === "admin") {
        console.log("Access granted");
        next(); // Move to the next middleware or route handler
    } else {
        console.log("Access denied");
        res.status(403).send("You do not have access to this resource.");
    }
});

// Routes
app.get("/admin/dashboard", (req, res) => {
    res.send("Welcome to the admin dashboard!");
});

app.get("/public", (req, res) => {
    res.send("This is a public page.");
});

Internal Execution Flow

Here’s what happens step by step:

Incoming Request: A request hits the server.
Middleware Execution:
- The middleware checks the req.user.role.
- If role is "admin", it calls next() to pass control to the next middleware or route.
- If not, the middleware terminates the request by sending a 403 Forbidden response.
Route Handler: If next() is called, the relevant route handler (e.g., /admin/dashboard) executes.

Example Flow:

A user with role: "admin" requests /admin/dashboard.
- Middleware logs "Access granted" and calls next().
- Route handler sends "Welcome to the admin dashboard!".
A user with role: "user" requests /admin/dashboard.
- Middleware logs "Access denied" and sends "You do not have access to this resource".

Key Takeaways

Middlewares are like gatekeepers—they decide what happens before the main route logic runs.
Use next() to ensure the flow continues to the next middleware or route.
Always define critical middlewares before the routes to ensure they apply.
If you don’t call next() or send a response, the request will hang (timed out).

Understanding .env Files and the dotenv Library in Node.js

Nishanthan K — Mon, 16 Dec 2024 04:00:00 +0000

Environment configuration is a critical part of any application development process. This article explores the purpose of .env files, how the dotenv library works, and why Node.js cannot directly access .env files. We'll also delve into the internal architecture of the dotenv library to understand its functionality.

What is a `.env` File?

A .env file is a simple text file used to store environment-specific configurations, such as:

API keys
Database credentials
Port numbers

The file follows a key-value pair format, making it easy to define and manage variables:

PORT=3000
DB_USER=devuser
DB_PASS=devpassword

Common Use Cases

Local Development: Deby velopers use .env files to avoid hardcoding sensitive information in the codebase.
Testing Environments: Testing environments can have their own .env file with configurations that mimic production setups.

Note:

.env files are typically excluded from production deployments due to security concerns. Sensitive data should be managed through environment variables provided by the hosting infrastructure or secret management tools.

Why Node.js Can't Access .env Directly

Node.js assumes that environment variables are managed externally by the system or runtime tools. For instance, these variables can be set using:

Operating system commands (export VAR_NAME=value on Unix/Linux)
CI/CD pipelines
Cloud hosting platforms (e.g., AWS, Heroku, Vercel)

.env files, on the other hand, are a developer convention designed for convenience during development. Node.js does not include native functionality to read .env files. This is where the dotenv library comes into play.

What is the dotenv Library?

dotenv is a lightweight Node.js library that:
Reads the .env file from the project root.
Parses its content into key-value pairs.
Injects these values into the global process.env object, making them accessible throughout your application.
By doing so, dotenv acts as a bridge between .env files and Node.js applications, simplifying the management of environment-specific settings.

Internal Workflow of dotenv

Here's how the dotenv library works under the hood:

File Reading: The library uses Node.js's fs module to read the .env file.
Parsing: The content of the file is parsed line by line. Key-value pairs are extracted using regular expressions.
Validation: The library skips blank lines and lines beginning with # (comments).
Injection: The parsed key-value pairs are added to process.env. If a variable already exists in process.env, the library typically avoids overwriting it (configurable).

Here's a simplified view of the dotenv workflow:

const fs = require('fs');
const path = require('path');

function parseEnvFile(filePath) {
    const content = fs.readFileSync(filePath, 'utf-8');
    const lines = content.split('\n');

    lines.forEach(line => {
        if (line.trim() && !line.startsWith('#')) {
            const [key, value] = line.split('=');
            process.env[key.trim()] = value.trim();
        }
    });
}

parseEnvFile(path.resolve(__dirname, '.env'));

Usage Workflow

Development and Testing

Create a .env file in the root of your project:

PORT=3000
DB_USER=devuser
DB_PASS=devpassword

Install dotenv:

npm install dotenv

Load the .env file in your application:

require('dotenv').config();

console.log(process.env.PORT); // Outputs: 3000

Production

For production environments, it's best to use infrastructure-level tools to manage environment variables:

Cloud Platforms: AWS Systems Manager, Google Secret Manager, Azure Key Vault
Containerized Environments: Docker --env flag or Kubernetes Secrets
CI/CD Pipelines: Define environment variables in the build/deployment pipeline.

By avoiding .env files in production, you ensure that sensitive information stays secure.

Best Practices

Keep .env Files Out of Version Control: Add .env to your .gitignore file to avoid accidental commits.
Use Environment-Specific Files: Maintain separate .env files for development (.env.development), testing (.env.testing), and staging (.env.staging).
Validate Variables: Use libraries like joi or dotenv-safe to ensure required environment variables are defined.
Limit Access: Only include variables that are absolutely necessary for the environment.

Conclusion

Environment configuration is a cornerstone of modern application development. By leveraging .env files and the dotenv library, developers can simplify local development and testing workflows. However, for production, it's essential to adopt secure practices for managing sensitive information. With a solid understanding of these tools and techniques, you'll ensure your applications are both secure and maintainable.

DEV Community: Nishanthan K

Backpropagation in Deep Learning: A Complete, Intuitive, and Practical Guide

Introduction: Why Backpropagation Matters

Why this matters:

What a Neural Network Really Computes (Forward Pass Overview)

Why this matters for backprop:

The Core Idea Behind Backpropagation

How should each weight in the network change to reduce the final error?”

Understanding the Computational Graph

Why this graph matters:

Chain Rule: The Mathematical Engine of Backpropagation

Why the chain rule is needed

Applying the chain rule to layers

Step-by-Step Backpropagation Through a Single Neuron

Forward Pass (Single Neuron)

Backward Pass (Goal: compute gradients)

Conclusion

Neural Network — A Simple, Beginner-Friendly Overview

What is a Neural Network?

How Are the Values Estimated?

Why Neural Networks?

Layers in a Neural Network

Input Layer

Hidden Layers

Output Layer

The 5-Step Life Cycle of a Neuron (Applies to Every Layer Except Input)

Final Thoughts

How Neural Networks Learn – A Simple Guide to Machine Learning & Deep Learning

1. Intelligence, Patterns, and Models

2. The Core Idea: Model as an Equation

3. Loss Function — Measuring Mistakes

A loss function:

4. Gradient — Finding the Direction of Improvement

Intuitively:

5. Learning Rate — Controlling the Speed of Learning

6. Epoch — Repeating the Learning Process

7. Batch — Dividing the Dataset

Benefits of batching:

8. Weights and Bias — Making Real-World Sense

Example: Courier Charges

9. Neural Network — From One Equation to Many

Why use neural networks?

10. Activation Function — The Source of Non-Linearity

Activation functions:

Without activation functions:

11. Forward Propagation — Data Moving Through the Network

Important:

12. Backward Propagation — Learning From Mistakes

13. The Complete Training Cycle

14. Inference — Using the Trained Model

Final Thought

Understanding Java Constructors: Types, Execution, and Differences from Methods

Understanding Constructors in Java

What is a Constructor?

Execution of Constructors

Compile Time

Runtime

Types of Constructors

Constructor vs Method in Java

Summary

Brushing Up on React Hooks

React Hooks Summary

1. useState

2. useEffect

3. useMemo

4. React.memo vs useMemo

5. useCallback

6. useRef

7. useReducer

8. useLayoutEffect

9. useImperativeHandle

10. useDebugValue

How Java Stores Data in Memory: Writing Efficient Code with Primitives and Non-Primitives

How Java Stores Data in Memory — Primitives vs Non-Primitives

Primitive Types in Java

Non-Primitive (Reference) Types

How Java Stores Data in Memory

Stack Memory

Heap Memory

How Primitive Types Are Accessed

1. `useState`

2. `useEffect`

3. `useMemo`

4. `React.memo` vs `useMemo`

5. `useCallback`

6. `useRef`

7. `useReducer`

8. `useLayoutEffect`

9. `useImperativeHandle`

10. `useDebugValue`

Misconception: Using `byte`/`short` Saves Memory?

Why `byte` and `short` are less efficient:

What is a `.env` File?