DEV Community: Sivananda Panda

Naive RAG is dead. Here is what agentic retrieval actually looks like

Sivananda Panda — Fri, 17 Jul 2026 18:44:06 +0000

Retrieval stopped being a fixed pipeline you run once. It became a decision the agent makes, again and again, while it works. Here is how I got there, and what changed.

The pattern that everyone shipped in 2023

If you built a RAG system a couple of years ago, you probably built the same thing I did. Take your documents, chop them into fixed-size chunks, embed each chunk, drop the vectors into a database. At query time, embed the question, grab the top few nearest chunks, paste them into the prompt, and let the model answer.

It demos beautifully. You ask a question, it pulls the right paragraph, the answer looks great, everyone nods. Then it goes to real users and starts getting things quietly wrong.

I have watched this happen more than once. The frustrating part is that nothing throws an error. The retrieval runs, chunks come back, the model answers with total confidence. It is just answering from the wrong context, and you only find out when someone downstream notices the answer is off.

So it is worth being honest about why the 2023 pattern breaks, because the failures are not random.

Why it breaks

The first problem is that vector similarity is not the same as relevance. Two chunks can sit close together in embedding space and mean opposite things in practice. I saw this clearly on a system dealing with policy tiers, where a "Tier 1" rule and a "Tier 2" rule were nearly identical as text but had completely different consequences. The embedding could not tell them apart, so the retriever kept handing back the wrong one. Semantically close, operationally opposite.

The second problem is fragmentation. When you split on a fixed character count, a single fact often gets cut across two chunks. The top-k pulls one half, misses the other, and no reranker on earth can fix a fact that was never retrieved whole. I wrote a whole separate piece on why chunking is a modeling decision, not preprocessing, and this is the heart of it.

The third problem is the one people underestimate. Real user queries are messy, and a surprising share of them fall outside whatever distribution your embedding model was trained on. One analysis put it around 31 percent of real queries landing out of distribution, which lines up with a jump in retrieval failures. Your embeddings were tuned on clean text. Your users type fragments, typos, and half-formed questions. The gap shows up as bad retrieval.

None of these are exotic. They are the default behavior of the naive pattern, which is exactly why "just add RAG" so often disappoints.

The first real fix: stop trusting one signal

The first upgrade is to stop relying on a single retrieval signal. Dense vector search is good at meaning but bad at exact terms like names, codes, and identifiers. Keyword search is the opposite. So you run both, dense and sparse, and combine them. Then you add a reranker, usually a cross-encoder, that looks at the query and each candidate together and reorders them by actual relevance instead of raw vector distance.

That combination, hybrid retrieval plus reranking, fixes a large chunk of the "semantically close but wrong" failures. On my own systems it was the single biggest quality jump.

But here is the part I want to stress, because it is the part most people skip. You cannot improve retrieval you are not measuring. Before I trust any retrieval change, I build a small set of real questions with the chunks that should come back for each one, and I score retrieval on that set. Did the right chunk make the top-k or not. Without that, you are tuning by vibes, and every change feels like progress whether or not it is.

Where it becomes agentic

Hybrid retrieval is still a fixed pipeline though. Same steps, every query, in the same order. The real shift, the thing that actually killed naive RAG for me, is treating retrieval as a decision the agent makes rather than a step you always run.

Think about what a careful researcher does. They do not embed your exact question and grab three paragraphs. They rephrase the question into something searchable. They decide whether they even need to look something up. If the first search is thin, they try a different angle. For a complex question they break it into parts and chase each one.

Agentic retrieval is that behavior, wired into the loop. A few concrete pieces of it:

Query rewriting comes first. The user's raw question is often a bad search query, so the agent rewrites it into one or more cleaner queries before anything hits the index. This alone handles a lot of that out-of-distribution problem.

Then there is the decision of whether to retrieve at all. Not every step needs a lookup. An agent that retrieves on every turn wastes tokens and drags in noise. A better one asks whether it already has what it needs.

For hard questions, the agent plans. It splits a multi-part question into sub-questions, retrieves for each, and combines what comes back. This is where single-shot RAG falls apart and an agent loop shines, because the answer to one sub-question shapes the next search.

And when a search comes back weak, the agent can notice and try again with a different query or a different source, instead of confidently answering from thin evidence.

The mental model that stuck with me is that retrieval turned into a policy, something the agent decides based on the state it is in, rather than a fixed function you call once and trust.

Knowing when not to retrieve at all

A quick but important detour, because the industry swung to a new extreme. With million-token context windows, plenty of people now say retrieval is pointless, just put everything in the prompt.

Sometimes that is right. If the whole relevant input is small and the model needs to reason across all of it at once, long context is simpler and often better. But every token in that window is paid for on every call, so at scale it gets expensive fast, and quality can drop as key details get buried in a huge prompt. Measured comparisons have put retrieval anywhere from roughly 8 to 82 times cheaper than stuffing an equivalent corpus into context, depending on the workload.

So it is not RAG or long context. Long context for small, nuanced inputs. Retrieval for large or fast-changing knowledge. Both together for multi-hop agent reasoning, where you retrieve to narrow things down and use context to reason over what came back.

Making the answer auditable

One more piece that separates a toy from something you can put in front of users. The agent should not just answer, it should show its work. Every claim in the output should point back to the chunk it came from.

This does two things. It lets a human verify the answer instead of trusting it, and it gives you a signal when retrieval was weak, because an answer with no solid source behind it is a red flag you can actually catch. Grounding and citation enforcement is not a nice-to-have. On a system where being wrong has a real cost, it is the difference between a demo and a tool.

What this looked like in practice

When I built my Research Investigation Agent, this is roughly the shape it took. It is a small graph of steps rather than a single call. The question first gets interpreted and planned. Retrieval is hybrid, pulling from both web search and a vector store, so it is not leaning on one signal. There is a self-reflection step where the agent checks whether the evidence it gathered is actually enough, and if it is not, it goes back and retrieves again before writing anything. And the whole run is traced, so I can see which step made which decision.

None of the individual ideas there are mine. What changed for me was seeing them fit together into one thing: retrieval as an ongoing decision inside a loop, checked and repeated, not a single hopeful lookup at the start.

The short version

Naive RAG is not dead because retrieval stopped mattering. It is dead because "embed, search top-k, paste, hope" was always the weakest possible version of retrieval. What replaced it is closer to how a person actually researches. Rephrase the question. Decide whether to look. Search more than one way. Check whether the evidence holds up. Try again if it does not. Show where the answer came from.

If your RAG system feels like it is guessing, it probably is. The fix is usually not a bigger model or a longer prompt. It is giving retrieval the ability to think.

What does your retrieval stack look like right now, a fixed pipeline or a loop that can decide?

Sources

Vector similarity vs relevance, and RAG failure modes in practice: https://towardsai.net/p/machine-learning/5-things-that-broke-my-rag-system
Chunking recall gap between strategies (~9 percent on the same corpus): https://ragflow.io/blog/rag-review-2025-from-rag-to-context
Out-of-distribution queries (~31 percent) and retrieval failures: https://ragaboutit.com/7-zero-shot-rag-failures-that-cost-enterprises-millions/
RAG vs long context cost gap (roughly 8 to 82x): https://byteiota.com/rag-vs-long-context-2026-retrieval-debate/ ; https://www.llamaindex.ai/blog/rag-is-dead-long-live-agentic-retrieval

Neural Networks From Scratch: The Math, and How It Becomes Code

Sivananda Panda — Sat, 11 Jul 2026 06:46:20 +0000

A beginner-friendly walk through how a neural network actually works under the hood, built in plain NumPy with no deep learning library. Every equation is followed by the code that implements it.

Full code: https://github.com/siva-rgb/Neural_Network_Scratch

Why I wrote this

When most of us start with machine learning, the path looks the same. You learn how an algorithm works in theory, you implement it with a library like scikit-learn, TensorFlow, or PyTorch, you pick up a few optimization and regularization tricks, and you move on to the next algorithm. I learned it that way too.

But neural networks always nagged at me. The library made them feel like a black box. Input goes into a hidden layer, weights and biases do something, that repeats until the final layer, and then some backpropagation step updates all the weights using gradient descent. It worked, but I could not have told you what was really happening between model.fit() and the accuracy going up.

So I sat down and rebuilt a neural network from scratch, using nothing but NumPy for the array math. A lecture series by Dr. Raj Dandekar helped a lot along the way. This article is the result: the theory and the math behind a neural network, translated line by line into Python. By the end, the black box should feel like a glass box.

First, the easy way

Here is a small neural network in TensorFlow, so we have something to compare against.

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# 1. Generate dummy data (1000 samples, 20 features)
X_train = np.random.random((1000, 20))
y_train = np.random.randint(2, size=(1000, 1))  # binary labels (0 or 1)

# 2. Build the network
model = Sequential([
    Dense(64, activation='relu', input_shape=(20,)),  # first hidden layer
    Dense(32, activation='relu'),                      # second hidden layer
    Dense(1,  activation='sigmoid')                    # output layer
])

# 3. Compile
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# 4. Train
model.fit(X_train, y_train, epochs=5, batch_size=32)

This is easy to read. Sequential stacks the layers, each Dense sets the number of neurons and the activation, compile picks the optimizer and the loss, and fit trains for 5 epochs in batches of 32. Four steps and you have a working network.

But that is exactly the problem. Every hard question, how the forward pass is computed, what the loss really measures, how backpropagation finds better weights, is hidden inside those four calls. So let us throw the library away and build all of it ourselves.

What a neural network actually is

Once I actually dug into it, a neural network turned out to be just two operations, repeated over and over:

A linear step: multiply the inputs by some weights, add a bias.
A non-linear step: pass the result through an activation function.

One round of those two is a layer. Stack a handful of them and you have got a deep network. And training? That is just nudging all those weights and biases, again and again, until the output starts matching the answer you actually wanted.

That is really the whole idea. Everything from here is just making those two steps precise and then writing them in code.

We will use a small, concrete example throughout: a fully connected network with 4 input features and a hidden layer of 4 neurons.

The forward pass: the math

Our single input example is a vector of four features:

X = [x_{1}, x_{2}, x_{3}, x_{4}] shape 1 \times 4

Every one of the 4 neurons has its own set of 4 weights, one per input, so the weights form a matrix:

W = ⎝ ⎛ w_{11} w_{21} w_{31} w_{41} w_{12} w_{22} w_{32} w_{42} w_{13} w_{23} w_{33} w_{43} w_{14} w_{24} w_{34} w_{44} ⎠ ⎞ shape 4 \times 4

Each neuron also has one bias:

b = [b_{1}, b_{2}, b_{3}, b_{4}]

The output of the layer, before any activation, is:

Z = X \cdot W^{T} + b

Read that in plain words: for each neuron, multiply every input by its matching weight, add them up, then add the neuron's bias. That single number is the neuron's pre-activation output. Do it for all four neurons and you get a vector of four values.

The forward pass: the code

Here is the exact same computation in NumPy.

import numpy as np

x_input = [2, 3, 4, 5]
w = [[0.3,  0.34, 0.4,  0.03],
     [0.5,  0.2,  0.1,  0.22],
     [0.23, 0.37, 0.4,  0.3 ],
     [0.02, 0.14, 0.17, 0.05]]
b = [2, -0.3, 5, 7]

dot_product = np.dot(np.array(w), np.array(x_input)) + np.array(b)
print(dot_product)
# [5.37  2.8   9.67  8.39]

One thing that tripped me up at first: we did not actually write a transpose here, even though the math clearly said $W^{T}$ . That is because NumPy treats a 1D array like x_input as a column vector of shape (4,). So np.dot(W, x) already computes, for each row of W (each neuron), the dot product with the input. The result is the same four numbers. When we move to batches of inputs later, the shapes change and the convention flips to X · W, which is why the reusable class below multiplies in that order. Shapes are the thing to keep your eye on in this whole exercise.

To prove there is no magic in np.dot, here is the same thing with plain Python loops:

layer_output = []
for weight_row, bias in zip(w, b):
    output = 0
    for n_input, weight in zip(x_input, weight_row):
        output += n_input * weight     # weighted sum
    output += bias                     # add the bias
    layer_output.append(output)

print(layer_output)
# [5.37, 2.8000000000000003, 9.67, 8.39]

Identical result. The loop makes the math obvious, but it is slow: its time complexity is roughly O(inputs times neurons), and it gets worse fast as layers grow. NumPy does the same arithmetic in optimized C under the hood, so we use it for everything from here on.

Finally, let us wrap the forward pass in a reusable class, the same idea as a Keras Dense layer:

class Layer_Dense:
    def __init__(self, n_input, n_neurons):
        # small random weights break symmetry so neurons learn different things
        self.weights = 0.01 * np.random.randn(n_input, n_neurons)
        self.bias    = np.zeros((1, n_neurons))

    def forward(self, inputs):
        self.output = np.dot(inputs, self.weights) + self.bias

Two small but important choices. Weights start as small random numbers so that no two neurons are identical at the start, if they all started equal they would stay equal and the network could never learn. Biases start at zero, which is fine.

With this, the forward pass of one layer is done, and every hidden layer works the same way.

Activation functions: giving the network its power

If we only ever stacked linear steps ( $X \cdot W + b$ ), the whole network would collapse into a single linear function, no matter how many layers. It could never learn a curve. Activation functions add the non-linearity that lets networks model complicated patterns.

ReLU, for hidden layers

The most common activation for hidden layers is ReLU, the Rectified Linear Unit. It is almost embarrassingly simple:

ReLU (x) = max (0, x)

Anything negative becomes zero, anything positive passes through unchanged.

class Activation_ReLU:
    def forward(self, inputs):
        self.output = np.maximum(0, inputs)

Softmax, for the output layer

For the final layer of a multi-class classifier we want probabilities that sum to 1. That is what softmax gives us. It exponentiates each value and then normalizes:

Softmax (x_{i}) = \frac{e ^{x_{i}}}{\sum _{j} e ^{x_{j}}}

There is one practical trick. Before exponentiating, we subtract the largest value in each row. This does not change the result mathematically (the ratio is identical), but it keeps the exponentials from blowing up to huge numbers and overflowing. This is called the numerical-stability trick.

class Activation_Softmax:
    def forward(self, inputs):
        # subtract the row max for numerical stability
        exp_values = np.exp(inputs - np.max(inputs, axis=1, keepdims=True))
        probabilities = exp_values / np.sum(exp_values, axis=1, keepdims=True)
        self.output = probabilities

For binary classification you would use the sigmoid function instead, which squashes a single value into the range 0 to 1. Here we will stick with softmax for the multi-class case.

The loss: measuring how wrong we are

Once the network produces probabilities, we need one number that says how far off it was. For classification we use categorical cross-entropy. For a single sample it is:

L = - lo g (p_{correct})

where $p_{correct}$ is the probability the network assigned to the true class. If the network was confident and right (say 0.99), the loss is tiny. If it was confident and wrong (say 0.01 for the true class), the loss is large. That is exactly the behaviour we want.

class Loss:
    def calculate(self, output, y):
        sample_losses = self.forward(output, y)
        return np.mean(sample_losses)   # average loss over the batch


class Loss_CategoricalCrossentropy(Loss):
    def forward(self, y_pred, y_true):
        samples = len(y_pred)
        # clip to avoid log(0), which is infinity
        y_pred_clipped = np.clip(y_pred, 1e-7, 1 - 1e-7)

        if len(y_true.shape) == 1:                    # sparse labels
            correct_confidences = y_pred_clipped[range(samples), y_true]
        elif len(y_true.shape) == 2:                  # one-hot labels
            correct_confidences = np.sum(y_pred_clipped * y_true, axis=1)

        return -np.log(correct_confidences)

What the shape == 1 vs shape == 2 check is doing. This is the part that confuses people, so let us be explicit. Labels can arrive in two formats:

Sparse (1D), shape == 1: the labels are just the class indices, for example [0, 2, 1] meaning sample 0 is class 0, sample 1 is class 2, sample 2 is class 1. To grab the predicted probability of the correct class we index directly: y_pred_clipped[range(samples), y_true] picks one probability per row.
One-hot (2D), shape == 2: the labels are vectors like [[1,0,0], [0,0,1], [0,1,0]]. Here we multiply element-wise by the predictions and sum across the row, which keeps only the probability at the "1" position.

Both paths end up with the same thing, the probability the network gave to the right answer. We also clip the predictions to a tiny range so we never take log(0), which would be negative infinity.

Backpropagation: how the network learns

This is the step that felt most like magic to me, so let us go slowly.

Forward propagation gave us a loss. Backpropagation answers a single question: if I nudge each weight and bias a little, how does the loss change? That answer is the gradient. Once we know it, we move each parameter in the direction that reduces the loss. Repeat thousands of times and the network gets better.

The tool behind it is the chain rule from calculus, and honestly, once that clicked for me it stopped being scary. The loss depends on the output, the output depends on the activation, the activation depends on the weights, and so on down the chain. The chain rule just lets us multiply all those little local derivatives together and push the error backward, from the loss all the way to the first layer.

The guiding principle for the code is simple and worth stating clearly:

Every operation that has a forward pass also has a backward pass. If the forward step computed an output from an input, the backward step computes the gradient of the loss with respect to that input.

Let us work through each piece.

The dense layer, backward

For the linear step $Z = X \cdot W^{T} + b$ , calculus gives three gradients:

\frac{\partial L}{\partial W} = X^{T} \cdot \frac{\partial L}{\partial Z}

\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Z} \cdot W^{T}

\frac{\partial L}{\partial b} = \sum \frac{\partial L}{\partial Z}

The gradient with respect to the weights tells us how to update them. The gradient with respect to the input is what we pass to the previous layer, so it can do its own update. The bias gradient is just the sum of the incoming gradients.

class Layer_Dense_v1:
    def __init__(self, n_inputs, n_neurons):
        self.weights = 0.01 * np.random.randn(n_inputs, n_neurons)
        self.bias    = np.zeros((1, n_neurons))

    def forward(self, inputs):
        self.inputs = inputs                       # remember inputs for the backward pass
        self.output = np.dot(inputs, self.weights) + self.bias

    def backward(self, dvalues):
        self.dweights = np.dot(self.inputs.T, dvalues)          # dL/dW
        self.dbias    = np.sum(dvalues, axis=0, keepdims=True)  # dL/db
        self.dinputs  = np.dot(dvalues, self.weights.T)         # dL/dX -> goes to previous layer

Notice we store self.inputs during the forward pass. The backward pass needs them, which is why frameworks keep a record of the forward computation.

ReLU, backward

The derivative of ReLU is beautifully simple: it is 1 wherever the input was positive, and 0 wherever it was negative or zero. So the backward pass just lets the gradient through where the input was positive and blocks it everywhere else.

class Activation_ReLU:
    def forward(self, X):
        self.input  = X
        self.output = np.maximum(0, X)

    def backward(self, dvalue):
        self.dinputs = dvalue.copy()
        self.dinputs[self.input <= 0] = 0     # gradient is 0 where input was not positive

Softmax and cross-entropy, backward, together

You could compute the derivative of the loss with respect to the softmax output, and then the derivative of softmax with respect to its input, and multiply them. But that is expensive and messy, because the softmax derivative is a full matrix (the Jacobian). It turns out that when softmax is paired with cross-entropy loss, almost everything cancels, and the combined gradient collapses to something wonderfully clean:

\frac{\partial L}{\partial Z} = y^- y

That is it. The predicted probabilities minus the true labels. This is why the two are almost always implemented together.

class Activation_Softmax_Loss_CategoricalCrossentropy:
    def __init__(self):
        self.activation = Activation_Softmax()
        self.loss       = Loss_CategoricalCrossentropy()

    def forward(self, inputs, y_true):
        self.activation.forward(inputs)
        self.output = self.activation.output
        return self.loss.calculate(self.output, y_true)

    def backward(self, dvalues, y_true):
        samples = len(dvalues)
        # if labels are one-hot, convert to class indices
        if len(y_true.shape) == 2:
            y_true = np.argmax(y_true, axis=1)

        self.dinputs = dvalues.copy()
        self.dinputs[range(samples), y_true] -= 1   # subtract 1 from the true class: (y_pred - y_true)
        self.dinputs = self.dinputs / samples       # average over the batch

Look at what self.dinputs[range(samples), y_true] -= 1 does. Starting from the predicted probabilities, it subtracts 1 from the entry of the correct class. That is exactly $y^- y$ in one line, because the one-hot true label is 1 only at the correct class. Then we divide by the number of samples to get the average gradient.

The optimizer: turning gradients into updates

We now have the gradients. The optimizer decides how to use them. The simplest one is plain gradient descent: move each parameter a small step in the opposite direction of its gradient.

W \leftarrow W - α \cdot \frac{\partial L}{\partial W}

where $α$ is the learning rate, how big a step we take.

class Optimizer_GD:
    def __init__(self, learning_rate=1.0):
        self.learning_rate = learning_rate

    def update_params(self, layer):
        layer.weights -= self.learning_rate * layer.dweights
        layer.bias    -= self.learning_rate * layer.dbias

Adding learning-rate decay

A fixed learning rate is a compromise. Too large and you overshoot the minimum and bounce around. Too small and training crawls. Decay gives you both: start with big steps to cover ground quickly, then shrink the steps over time to settle precisely into the minimum.

α_{t} = \frac{α _{0}}{1 + decay \cdot t}

where $t$ is the iteration number.

class Optimizer_GD_with_decay:
    def __init__(self, learning_rate=1.0, decay=0.0):
        self.learning_rate         = learning_rate
        self.current_learning_rate = learning_rate
        self.decay                 = decay
        self.iterations            = 0

    def pre_update_params(self):
        if self.decay:
            self.iterations += 1
            self.current_learning_rate = self.learning_rate * (1. / (1. + self.decay * self.iterations))

    def update_params(self, layer):
        layer.weights -= self.current_learning_rate * layer.dweights
        layer.bias    -= self.current_learning_rate * layer.dbias

The benefit over a plain learning rate is faster early progress with a more stable, precise finish. The shortcoming is that decay is blind to the actual terrain of the loss. It shrinks the step on a schedule whether or not that is the right thing to do at that moment, and it can slow down too early on hard problems.

Adding momentum

Momentum fixes some of that blindness by borrowing an idea from physics. Instead of stepping purely on the current gradient, it keeps a running velocity, a memory of the recent gradients, and moves along that. When gradients keep pointing the same way, momentum builds speed. When they oscillate, momentum cancels the wobble out.

v_{t} = μ \cdot v_{t - 1} - α \cdot \frac{\partial L}{\partial W}, W \leftarrow W + v_{t}

where $μ$ is the momentum factor.

class Optimizer_GD_with_momentum:
    def __init__(self, learning_rate=1.0, decay=0.0, momentum=0.0):
        self.learning_rate         = learning_rate
        self.current_learning_rate = learning_rate
        self.decay                 = decay
        self.iterations            = 0
        self.momentum              = momentum

    def pre_update_params(self):
        if self.decay:
            self.iterations += 1
            self.current_learning_rate = self.learning_rate * (1. / (1. + self.decay * self.iterations))

    def update_params(self, layer):
        if self.momentum:
            if not hasattr(layer, 'weight_momentums'):
                layer.weight_momentums = np.zeros_like(layer.weights)
                layer.bias_momentums   = np.zeros_like(layer.bias)

            weight_updates = self.momentum * layer.weight_momentums - self.current_learning_rate * layer.dweights
            layer.weight_momentums = weight_updates

            bias_updates = self.momentum * layer.bias_momentums - self.current_learning_rate * layer.dbias
            layer.bias_momentums = bias_updates
        else:
            weight_updates = -self.current_learning_rate * layer.dweights
            bias_updates   = -self.current_learning_rate * layer.dbias

        layer.weights += weight_updates
        layer.bias    += bias_updates

The benefit over plain decay is that momentum accelerates through long, consistent slopes and dampens oscillation in narrow valleys, which usually means faster and steadier convergence. It can also carry the network across small bumps and saddle points where plain gradient descent would stall. The cost is one more hyperparameter to tune, and if you set momentum too high it can overshoot and become unstable. This same line of thinking, adaptive per-parameter steps, is what leads to optimizers like Adam.

Putting it all together

Here is the full training loop, using everything we built. Two dense layers, a ReLU in the middle, and the combined softmax-plus-cross-entropy at the end.

dense1      = Layer_Dense_v1(2, 64)      # 2 inputs -> 64 neurons
activation1 = Activation_ReLU()
dense2      = Layer_Dense_v1(64, 3)      # 64 -> 3 output classes
loss_act    = Activation_Softmax_Loss_CategoricalCrossentropy()
optimizer   = Optimizer_GD()

for epoch in range(10000):
    # ---- forward pass ----
    dense1.forward(X)
    activation1.forward(dense1.output)
    dense2.forward(activation1.output)
    loss = loss_act.forward(dense2.output, y)

    # ---- track accuracy ----
    predictions = np.argmax(loss_act.output, axis=1)
    if len(y.shape) == 2:
        y = np.argmax(y, axis=1)
    accuracy = np.mean(predictions == y)
    if not epoch % 100:
        print(f'epoch: {epoch}, acc: {accuracy:.3f}, loss: {loss:.3f}')

    # ---- backward pass ----
    loss_act.backward(loss_act.output, y)
    dense2.backward(loss_act.dinputs)
    activation1.backward(dense2.dinputs)
    dense1.backward(activation1.dinputs)

    # ---- update ----
    optimizer.update_params(dense1)
    optimizer.update_params(dense2)

Read the loop top to bottom and you can see the entire story of a neural network in one screen. Data flows forward through the layers to a loss. The gradient flows backward through the exact same layers in reverse order. The optimizer nudges every weight and bias. Then it all repeats, and with each pass the accuracy climbs and the loss falls.

What I took away from this

Building this from scratch changed how I see every deep learning model. A Dense layer is a weighted sum plus a bias. An activation is a small non-linear function with an equally small derivative. A loss is one honest number. Backpropagation is the chain rule applied carefully, backward. And an optimizer is just a rule for taking a step.

Once those five pieces click, the bigger architectures stop being intimidating. A CNN, an RNN, even a transformer are the same ideas rearranged. The box is no longer black.

If you want to run it yourself, the full implementation is here:
https://github.com/siva-rgb/Neural_Network_Scratch

And if you want the deeper theory and full implementation, Dr. Raj Dandekar's Video Lecture lecture series is well worth your time.

The Evaluation Layer: The Part of Your LLM System You Keep Skipping

Sivananda Panda — Thu, 02 Jul 2026 11:43:52 +0000

I've built two agentic AI systems over the past few months, and despite solving very different problems, both exposed the same weakness. The agents worked perfectly during demos. They passed my manual tests. They looked production-ready. But once real users started interacting with them, confidently wrong responses began slipping through. The root cause wasn't the model, the prompts, or the tools. It was much simpler: there was no proper evaluation layer. The system could tell whether it had produced an answer, but it had no way to determine whether that answer was actually good.

This article is about the layer that closes that gap. What it is, why LLM systems specifically can't survive without it, the five ways people actually build it, the tooling, and the mistakes that make an eval layer worse than useless — because a miscalibrated eval gives you false confidence, which is more dangerous than no eval at all.

I'll use a RAG-and-agent stack for the concrete examples (LangGraph, Claude, LangSmith), but nothing here is framework-specific. The principles move to any stack you like.

Why LLM systems need this and normal software doesn't

Here's the uncomfortable property that breaks every habit you brought from traditional engineering: the same input can produce different outputs, and a wrong output usually looks exactly like a right one.

In a normal codebase, add(2, 2) returns 4 or it returns a bug you can see. The failure has a shape — a stack trace, a null, a red test. You write an assertion, it passes forever, you move on. LLM failures don't have that shape. A hallucinated citation, a subtly-off summary, a tool call with the wrong argument that still "sort of" works — these render as fluent, plausible, professional-looking text. The failure is camouflaged inside a success.

Three things follow from that, and they're the whole reason the eval layer exists:

Non-determinism means one passing run proves nothing. You need distributions, not point checks.
The interesting failures live in the middle. An agent that made 12 tool calls and 4 reasoning hops can reach a correct answer through completely broken logic — and reach a wrong one next time from the same code.
Nobody sees the middle by default. The end user gets the final answer. The reasoning, the retrieval, the tool arguments — all invisible unless you deliberately capture them.

So the eval layer isn't a testing afterthought. It's the observability and quality infrastructure that lets you answer three questions on a continuous basis: is the system doing the right thing (correctness), doing it efficiently (cost and latency), and doing it safely (no leaked secrets, no disallowed tools, no confident nonsense). Skip it and you're not shipping a product; you're running an ongoing experiment on your users without reading the results.

What the layer actually is

It's tempting to picture "the eval layer" as one service you bolt on at the end. It isn't. It's a cross-cutting concern that taps into every stage of execution and scores the trace, not just the output.

Three moving parts. Tracers capture what happened at every step — inputs, outputs, tool arguments, retrieved chunks, latencies, token counts. Scorers turn those captured artifacts into numbers — a faithfulness score, a latency measurement, a pass/fail on a schema. Evals are the curated sets and thresholds that give those numbers meaning: is 0.81 faithfulness good, and is it better or worse than last week? Lose any one of the three and the other two stop being useful.

Five ways to build it

These aren't competing options where you pick one. A mature system runs all five, at different frequencies. But you'll add them in roughly this order, cheapest and most objective first.

1. Deterministic evals — start here, not with an LLM

The instinct is to reach for an LLM judge immediately because the outputs are "fuzzy." Resist it. A surprising amount of what you care about is not fuzzy at all, and plain code checks it faster, cheaper, and without any of the reliability problems a judge brings.

Did the agent call only allowed tools? Did the structured output match the schema? Did it stay under the tool-call budget? Did the JSON parse? These have crisp right answers, and a regular function is the correct instrument.

def eval_tool_call_validity(trace: AgentTrace) -> EvalResult:
    """Every tool call must use an allowed tool and respect the budget."""
    allowed_tools = {"search", "calculator", "fetch_document"}
    violations = []

    for step in trace.steps:
        if step.type != "tool_call":
            continue
        if step.tool_name not in allowed_tools:
            violations.append(f"Disallowed tool: {step.tool_name}")
        if step.call_count > MAX_TOOL_CALLS:
            violations.append(f"Exceeded call budget: {step.call_count}")

    return EvalResult(
        passed=not violations,
        score=1.0 if not violations else 0.0,
        details=violations,
    )

These cost microseconds and never disagree with themselves. Run them on every single trace in production, not just in CI. They're your smoke detectors.

2. LLM-as-judge — powerful, and the thing most likely to lie to you

For the genuinely subjective stuff — tone, helpfulness, whether an answer is faithful to its source — you hand the output to a separate model with a rubric and collect structured scores.

from anthropic import Anthropic
import json

client = Anthropic()

JUDGE_PROMPT = """You are evaluating an AI agent's response.

User query: {query}
Agent response: {response}
Retrieved context: {context}

Score each dimension 1-5, and cite the specific text that justifies your score.
- Faithfulness: Is every claim grounded in the provided context?
- Relevance:    Does it address what the user actually asked?
- Completeness: Does it fully answer the question?

Respond ONLY as JSON:
{{"faithfulness": N, "relevance": N, "completeness": N, "evidence": "...", "reasoning": "..."}}"""

def llm_judge(query: str, response: str, context: str) -> dict:
    result = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=600,
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(query=query, response=response, context=context),
        }],
    )
    return json.loads(result.content[0].text)

Notice I made the judge cite evidence and reason, not just emit a number. That's not decoration. A bare score is unfalsifiable; a score with quoted evidence can be audited when you disagree with it.

Here's the part teams underrate: LLM judges have the same weaknesses as the model being judged. They reward length. They're swayed by confident phrasing. They wave through fluent errors — the exact failure mode you built the judge to catch. An uncalibrated judge doesn't measure quality, it launders your existing biases into an official-looking number. So the judge is not the end of the work; it's a component that itself needs validating against human labels before you trust a word it says. More on that in the mistakes section, because it's the one people get wrong most often.

3. Trace-level evaluation — stop grading the black box

Grading only the final answer is like reviewing a math test by checking the last number and ignoring the working. The number can be right for the wrong reasons and wrong for the same reasons next time.

The fix is to instrument every step and evaluate the whole trace. Observability platforms — LangSmith, Langfuse, Arize Phoenix, W&B Weave — exist for exactly this. You wrap your nodes and they capture the tree of execution:

from langsmith import traceable, Client

ls = Client()

@traceable(run_type="chain", name="research_agent")
def run_agent(query: str) -> str:
    plan   = planner.plan(query)          # captured as a child run
    docs   = retriever.fetch(plan.query)  # captured as a child run
    answer = generator.generate(docs)     # captured as a child run
    return answer

def score_run(run_id: str, scores: dict):
    ls.create_feedback(
        run_id=run_id,
        key="faithfulness",
        score=scores["faithfulness"],
        comment=scores["reasoning"],
    )

The payoff is diagnostic, not just descriptive. When faithfulness tanks, the trace tells you where: the retriever pulled garbage, or the retriever was fine and the generator ignored it. Those are two completely different bugs with two completely different fixes, and output-only evals can't tell them apart.

4. Human-in-the-loop — the ground truth everything else calibrates against

Automated evals are necessary and insufficient. Humans catch what code and judges structurally cannot: domain inaccuracies a generalist judge waves through, tone that's wrong for your specific users, edge cases that are technically correct and practically useless. And critically — human labels are the yardstick you use to check whether your LLM judge is any good. Without them, every other layer is measuring against itself.

Four things separate a useful human-eval pipeline from a box-ticking one:

Sample successes, not just failures. "The agent returned an answer" is not evidence the answer was good. Review 5–10% of all production traces weekly, including the ones nobody complained about.
Stratify the sample. Cover query types, user segments, and tool paths. Review only the easy queries and you'll build a rosy, useless picture.
Write annotation rubrics that force a decision. "Was this response good?" produces noise. "Did the response answer the user's stated question without inventing information they didn't ask for?" produces labels you can act on.
Measure agreement between annotators. Run the same trace past two reviewers now and then. If they disagree more than one time in five, the problem is your rubric, not your reviewers — sharpen it before you collect more labels.

5. Regression and red-teaming — so tomorrow's fix doesn't quietly break today's feature

Every prompt tweak is a potential regression, and LLM regressions are invisible without a locked baseline to compare against. A regression suite is a curated set of (input, expected) pairs you run on every deploy. Red-teaming is its adversarial twin: inputs deliberately built to break things — prompt injection, context-stuffing, multi-hop manipulation, tool misuse. In agentic systems these are nastier than in a single call, because one poisoned step can cascade through every step that follows it.

evals/
  regression/
    core_qa.jsonl          # 50 representative Q&A pairs
    tool_use.jsonl         # 30 traces exercising tool-call patterns
    edge_cases.jsonl       # 20 known-difficult inputs
    adversarial.jsonl      # 15 red-team prompts
  run_suite.py             # runner that diffs scores against baseline
  baseline_scores.json     # locked scores from the last known-good release

Run it in CI before every deploy. Alert when any category drops more than ~5% from baseline. The exact number matters less than the fact that there is one and something screams when you cross it.

Six mistakes that turn an eval layer into a liability

An eval layer isn't automatically good. A bad one is worse than none, because "our scores are green" is a sentence that stops people from looking closer. These are the six failure modes I see most.

Grading only the final output. Covered above, but it's the number-one blind spot so it earns repeating: a correct answer reached through a broken trace is a landmine, not a pass. Evaluate every node, not just the terminal one.

Collapsing everything into one score. A composite of 0.72 tells you something is wrong and nothing about what. Is it faithfulness? Latency? Tool selection? Keep the dimensions separate, track them separately, set thresholds separately. You can always average later; you can't un-average.

Evaluating on your training distribution. If your eval set is made from the same queries you used to build and tune the system, you've optimized to the eval, not to reality — Goodhart's Law wearing a lab coat. Hold out a blind set that never touches development, and keep topping it up with real production samples. Treat eval-set contamination as seriously as data leakage.

Trusting an uncalibrated judge. The single most common self-inflicted wound. Before an LLM judge scores anything that matters, run it against 100+ human-labeled examples and compute how often it agrees with humans. Disagreement above ~15%? Fix the judge prompt before it goes anywhere near your dashboard. A judge you haven't validated is a confidence machine, not a measurement.

No baseline, no history. Scores you don't persist can't answer "better or worse than last week?" Every run should write to a durable store with a timestamp, run ID, and git SHA. A SQLite table is enough — the discipline of storing beats the sophistication of the storage.

Treating evals as a one-time setup. Eval suites rot. User behavior drifts, new failure modes appear, and a suite built in month one is riddled with blind spots by month six. Add a case from every production failure within a couple of days of finding it, and audit the whole suite quarterly. It's a living artifact, not a shipped deliverable.

Setting it up: a practical walkthrough

Opinionated defaults for a LangGraph production agent. Swap the tools freely; the sequence is the point.

Step 1 — Define "good" in prose before you write any eval code. For each capability, answer on paper: what does a correct output look like, what are the common failure modes, what constraints must always hold, and what separates "barely acceptable" from "excellent" in a domain expert's eyes? This costs half a day. Skip it and you'll spend weeks precisely measuring the wrong things.

Step 2 — Instrument the graph. Wrap every node. Capture node name, input, output, latency, token counts, and any tool calls with arguments and returns.

from langsmith import traceable

@traceable(run_type="llm", name="planner_node")
def planner_node(state: AgentState) -> AgentState:
    response = llm.invoke(state["messages"])
    return {"plan": response.content, **state}

@traceable(run_type="tool", name="retriever_node")
def retriever_node(state: AgentState) -> AgentState:
    docs = retriever.invoke(state["plan"])
    return {"context": docs, **state}

Step 3 — Build the suite in three tiers, by frequency.

Tier 1 — every trace, real time: schema validation, tool-call constraints, latency thresholds. Cheap deterministic code.
Tier 2 — daily or per release: LLM-judge scores on a sample, retrieval metrics (precision@k, NDCG), end-to-end correctness on the regression suite.
Tier 3 — weekly or per major release: stratified human review, red-team annotation, edge-case triage.

Step 4 — Set thresholds before production, not during the incident. For every metric, define a green zone (normal), a yellow zone (investigate, don't block), and a red zone (block the deploy or page, someone). "We'll know bad when we see it" is the sentence people say right before a bad week.

Step 5 — Wire it into CI/CD. An eval that only runs when someone remembers doesn't run.

# .github/workflows/eval.yml
name: Eval Suite
on: [push]
jobs:
  run-evals:
    steps:
      - name: Run regression suite
        run: python evals/run_suite.py --compare-to baseline_scores.json
      - name: Check score thresholds
        run: python evals/check_thresholds.py --fail-on-regression 5

What to measure and what to keep

Split metrics into three families and never blend them into one number.

Correctness

Metric	What it measures	How to compute
Faithfulness	Is every claim grounded in retrieved context, with nothing hallucinated?	LLM judge or NLI model comparing response to context
Answer relevance	Does the response address the actual question?	LLM judge or embedding similarity between query and response
Context precision	Of the chunks retrieved, what fraction were useful?	Human or LLM label per chunk
Context recall	Did retrieval surface everything needed to answer?	Compare retrieved set against a gold document set
Tool-call accuracy	Right tools, right arguments?	Deterministic diff against an expected tool trace

Efficiency

Metric	What it measures	Target
Latency (p50/p95/p99)	User-perceived speed	Track trends; set SLOs per use case
Token consumption	Cost per query	Input + output tokens per run
Tool-call count	Wasted calls	Compare to the minimum viable count
Retry rate	How often steps fail and rerun	Under ~5% in steady state
Context-window utilization	How full the window runs	High → truncation risk

Safety and reliability

Metric	What it measures
Hallucination rate	% of responses with claims unsupported by context
Refusal rate	% of valid queries wrongly refused
Task-completion rate	% of queries reaching a terminal answer
Error rate by type	Tool failures, timeouts, parse errors — broken out, not summed
Constraint-violation rate	% of runs breaking a defined rule (e.g. a disallowed tool)

And keep the artifacts, because a score with no trace behind it is impossible to debug. For every run, preserve: the full execution trace (at least 30 days of production), per-dimension scores linked to run ID and git SHA, the judge's reasoning (not just its number — this is gold when you contest a score), failure cases tagged by type, the versioned eval-suite definition, the current baseline snapshot, and human annotation logs with annotator ID and timestamp.

Picking your tooling

Tool	Best for	The catch
LangSmith	LangChain/LangGraph shops wanting tight integration	Vendor lock-in; price scales with trace volume
Langfuse	Open-source, self-hostable	More setup; smaller ecosystem
Arize Phoenix	Teams already on Arize for ML monitoring	Stronger on classic ML; newer for LLMs
W&B Weave	Teams already living in Weights & Biases	Natural fit if you also fine-tune
RAGAS	RAG metrics out of the box	Narrow scope — mostly retrieval + generation
Custom (SQLite + an SDK)	Maximum control, minimal dependency	You own the build and the maintenance

My honest default for a LangGraph production system: LangSmith for tracing, RAGAS for RAG-specific metrics, and a small custom Python runner for the deterministic checks. Add human-eval tooling — even a scrappy Streamlit annotation app — once the system is past its first real users. Don't buy the enterprise platform on day one; you don't yet know what you're measuring.

The actual mindset shift

The thing most teams get wrong isn't a tool choice. It's timing. They treat evaluation as a phase that comes after building, and by then the design decisions that would have made the system measurable are already baked in.

Flip it. Evaluation is a lens you hold up while building. When you write a new node, the first question isn't "does this code run?" — it's "how will I know if this node is doing the right thing next Tuesday, in production, on a query I haven't seen?" When you tune a prompt, you don't eyeball three examples and ship on a good feeling; you run the suite and read the diff.

That's the whole difference between a demo and a system you can put your name on. Without an eval layer you're steering on vibes, and vibes don't survive contact with real traffic. With one, every decision has evidence under it. Build it early, treat it as first-class engineering rather than QA cleanup, and never push a change to production without knowing what your scores say about it.

Checklist

Before first deploy

[ ] Eval criteria written down for each capability
[ ] Every node instrumented with tracing
[ ] Tier-1 unit evals live (schema, constraints, latency)
[ ] Regression suite built (50+ curated examples)
[ ] Green/yellow/red thresholds set for every metric
[ ] Regression suite wired into CI/CD
[ ] Baseline scores locked

Weekly

[ ] Human review of a random 5–10% production sample
[ ] Post-mortem on every red-zone incident
[ ] New failure cases folded into the suite

Quarterly

[ ] Suite audit — cut stale cases, close coverage gaps
[ ] Re-calibrate LLM judges against fresh human labels
[ ] Revisit thresholds — still the right lines?
[ ] Run a red-team exercise and act on what breaks

I Thought Dimensionality Reduction Belonged to Classical ML. Then It Changed How I Think About AI.

Sivananda Panda — Tue, 23 Jun 2026 06:48:57 +0000

In my previous article, I explored several dimensionality reduction techniques, including PCA, t-SNE, UMAP, LDA, Sammon Mapping, KNN Graphs, and Autoencoders.

Going into that project, my goal was fairly simple.

I wanted to understand how these algorithms worked, where they were useful, and whether they could improve model performance.

Like many people learning machine learning, I saw dimensionality reduction as a classical ML topic.

Something you learn alongside feature engineering and data preprocessing.

Useful knowledge, but not something I expected to connect to modern AI systems.

I was wrong.

Not because PCA powers Large Language Models.

Not because dimensionality reduction is secretly the most important topic in machine learning.

But because the project forced me to think about a question I hadn't considered before.

The Question I Didn't Expect

While comparing different algorithms, I noticed something strange.

The same dataset could look completely different depending on the technique I used.

PCA produced one view.

t-SNE produced another.

UMAP showed patterns that weren't obvious in either of them.

Autoencoders created their own representation altogether.

At first, I was focused on which visualization looked better.

Then I started asking a different question:

If all of these algorithms are looking at exactly the same data, what are they actually trying to preserve?

That question turned out to be far more interesting than finding the "best" dimensionality reduction algorithm.

PCA Isn't Really About Reducing Dimensions

Most tutorials introduce PCA as a way to reduce features.

For example, a dataset with 100 features might be compressed into 10 principal components while retaining most of the variance.

That explanation is correct.

But while experimenting with PCA, I realized I was focusing on the wrong part of the process.

The interesting part wasn't that 100 dimensions became 10.

The interesting part was that the transformed data still retained much of the structure of the original dataset.

Some information was discarded.

Some information was preserved.

The algorithm had effectively made a decision about what mattered.

And that idea kept showing up across other dimensionality reduction techniques.

Each algorithm compressed the data differently because each algorithm had a different definition of what should be preserved.

The Moment Things Started Clicking

The more I explored these techniques, the less I thought about dimensions and the more I thought about representations.

PCA preserves variance.

t-SNE focuses heavily on local neighborhoods.

UMAP attempts to maintain structural relationships.

Autoencoders learn their own compressed representation from data.

Different algorithms.

Different mathematics.

Different outputs.

But they all seemed to revolve around the same challenge:

How do you transform information into a form that still captures the important patterns?

Once I started thinking that way, I began noticing the same idea outside dimensionality reduction.

Why This Matters Beyond Classical Machine Learning

One thing that often happens when you're learning machine learning is that topics get placed into separate mental boxes.

Classical ML.

Deep Learning.

LLMs.

Recommendation Systems.

Computer Vision.

NLP.

They can feel like completely different worlds.

But sometimes the same ideas appear in all of them.

Take image classification.

A neural network doesn't look at an image the same way throughout the entire model.

Early layers respond to simple patterns.

Edges.

Textures.

Basic shapes.

Deeper layers work with increasingly abstract representations.

By the time the model makes a prediction, it is operating on something very different from the original pixels.

The representation has changed multiple times.

The model has transformed the data into a form that makes the task easier.

That sounded surprisingly familiar.

Then I Started Looking at LLMs

The same thing happened when I started learning more about embeddings.

Consider the words "car" and "vehicle."

They are different words, yet most embedding models place them relatively close together in vector space.

The model isn't storing dictionary definitions.

It is learning a representation that captures part of the relationship between those words.

The exact mechanism is very different from PCA.

The mathematics is different.

The scale is different.

But the underlying idea felt familiar.

Once again, information was being transformed into a representation that preserved what mattered for the task.

That was the connection I hadn't expected when I started learning dimensionality reduction.

A Better Mental Model

Many beginners think about machine learning like this:

Data → Model → Prediction

After working through these dimensionality reduction techniques, I think a more useful mental model is:

Data → Representation → Model → Prediction

Because the way information is represented often determines what patterns a model can learn.

Two systems can work with the same underlying data and arrive at different outcomes simply because they represent that data differently.

That's true for PCA.

It's true for Autoencoders.

It's true for embeddings.

And it's true for many modern AI systems.

Final Thoughts

I started learning dimensionality reduction because I wanted to understand PCA, t-SNE, UMAP, and a few other algorithms.

What I didn't expect was that those techniques would change how I think about machine learning itself.

The biggest lesson wasn't about reducing dimensions.

It wasn't about preprocessing.

And it wasn't about improving model accuracy.

It was realizing that many AI systems, regardless of how different they appear on the surface, spend a significant amount of effort answering the same question:

How should information be represented so that useful patterns become easier to discover?

For me, dimensionality reduction was the first place where that idea became visible.

And once I noticed it, I started seeing it everywhere.

I Compressed 784 Dimensions Into 2. Here's What 70,000 Handwritten Digits Actually Look Like

Sivananda Panda — Mon, 22 Jun 2026 08:59:01 +0000

PCA Didn't Improve My Model. It Changed How I Think About Data Instead.

When I ran PCA on a dataset I was exploring, I expected a fairly straightforward outcome.

Reduce dimensionality.

Remove noise.

Train the model again.

Get better performance.

That's the story dimensionality reduction is often associated with.

The reality was much less exciting.

The accuracy barely moved.

At first, I treated PCA as a failed experiment.

Looking back, the failed experiment was actually my mental model.

I had been focused on improving the model without understanding something more fundamental:

What did the data actually look like?

That question eventually led me into one of the most interesting machine learning rabbit holes I've explored so far.

The Problem I Didn't Realize I Had

Like most practitioners, I started with exploratory data analysis.

I checked distributions.

Looked for missing values.

Analyzed correlations.

Built baseline models.

Reviewed performance metrics.

All useful activities.

But none of them answered a question that suddenly felt important:

If every sample is represented as hundreds of features, what shape does this data actually have?

Machine learning models operate in spaces that humans can't visualize.

A dataset with 500 features exists in a 500-dimensional space.

A dataset with 1000 features exists in a 1000-dimensional space.

We can calculate distances in those spaces.

We can train models in those spaces.

But we can't intuitively understand them.

And yet many of the questions we care about are geometric in nature.

Are classes naturally separable?

Are there meaningful clusters?

Are there outliers?

Do the samples lie on some hidden structure?

The information already exists in the data.

The challenge is making it visible.

Discovering a Better Playground

While reading about dimensionality reduction, I came across Christopher Olah's brilliant article on visualizing the MNIST dataset.

For anyone unfamiliar with it, MNIST contains handwritten digits from 0 to 9.

Each image is only 28×28 pixels.

That sounds tiny.

But once flattened, every image becomes a point in a 784-dimensional space.

Each handwritten digit is represented by 784 numbers.

Humans can't visualize 784 dimensions.

Dimensionality reduction algorithms can help us project that space into something we can see.

What fascinated me wasn't the mathematics.

It was the possibility that different algorithms might reveal different aspects of the same dataset.

So I downloaded MNIST and started experimenting.

What began as curiosity quickly turned into a project.

I implemented and compared:

PCA
LDA
t-SNE
UMAP
Sammon Mapping
KNN Graph Visualizations
Autoencoders

My expectation was simple.

Different algorithms would produce slightly different versions of the same visualization.

I was completely wrong.

The Most Interesting Question

After generating the first set of visualizations, I found myself staring at the outputs, wondering:

Which one of these is actually correct?

PCA showed one picture.

t-SNE showed another.

UMAP showed something different again.

The Autoencoder latent space looked completely different from everything else.

They couldn't all be right.

Except they were.

The mistake was assuming they were trying to answer the same question.

Each algorithm was optimizing for a different definition of "important structure."

Once I understood that, dimensionality reduction became much more interesting.

What PCA Actually Taught Me

PCA was the first method I explored because it's usually the default recommendation.

The intuition is elegant.

Find the directions where the data varies the most and project onto those directions.

Simple.

Fast.

Interpretable.

What surprised me was that PCA didn't produce the clean digit separation I expected.

Some digits remained heavily mixed together.

Initially, this felt disappointing.

Then I realized PCA had taught me something important.

Variance is not the same thing as class separation.

The largest source of variation in a dataset isn't necessarily the information that distinguishes one class from another.

A thick handwritten "2" and a thin handwritten "2" can contribute substantial variance while still belonging to the same class.

PCA isn't trying to separate classes.

It's trying to preserve variance.

That distinction seems obvious in hindsight, but seeing it visually made the lesson stick.

What LDA Taught Me About Labels

The jump from PCA to LDA was dramatic.

Suddenly, the classes looked much cleaner.

The clusters became easier to distinguish.

The reason wasn't that LDA is universally superior.

The reason is that LDA has access to information PCA never sees.

Labels.

PCA asks:

Where is the variance?

LDA asks:

How can I maximize separation between known classes?

Those are fundamentally different objectives.

Running both methods side by side highlighted something important about machine learning in general.

The information contained in labels is incredibly valuable.

Once an algorithm knows what you want to separate, it can optimize directly for that objective.

Without labels, it has to infer structure on its own.

What t-SNE Taught Me About Beautiful Visualizations

The first visualization that made me stop and stare was t-SNE.

The clusters looked incredible.

Digits formed tight, well-separated groups.

The output was visually satisfying in a way PCA never was.

It almost looked as though the dataset had organized itself.

Then I started reading more about how t-SNE works.

That's when I learned an important lesson.

t-SNE prioritizes preserving local neighborhoods.

Points that are close together remain close together.

Global geometry becomes much less important.

This means something subtle but important.

The clusters themselves are often meaningful.

The distances between clusters are often not.

Humans naturally assume that if Cluster A is closer to Cluster B than Cluster C, then A and B must be more similar.

With t-SNE, that assumption can easily be wrong.

The experience taught me something I now apply beyond dimensionality reduction.

The most visually impressive result isn't always the most informative one.

Why UMAP Felt Different

After seeing the extremes of PCA and t-SNE, UMAP felt like the first algorithm that was trying to strike a balance rather than optimize a single objective.

PCA focuses on preserving variance.

t-SNE focuses heavily on preserving local neighborhoods.

UMAP sits somewhere in between.

The underlying assumption behind UMAP is that high-dimensional data often lies on a lower-dimensional manifold. Instead of asking where the variance is largest, UMAP asks a different question:

Which points genuinely belong together, and what hidden structure could explain those relationships?

For my experiments, I used 15 neighbors, a cosine distance metric, and projected the data into three dimensions. These choices turned out to be important.

Using 15 neighbors meant that each digit considered a reasonably sized local neighborhood when constructing the manifold. If I had chosen a much smaller value, the visualization would have focused almost entirely on local structure, producing tighter but potentially misleading clusters. A much larger value would have emphasized global relationships at the expense of local detail. Fifteen felt like a practical middle ground.

The cosine distance metric was equally interesting. Instead of measuring absolute pixel differences, cosine similarity focuses on whether two images share similar patterns. For handwritten digits, that matters. Two people can write the same digit with different stroke thicknesses and intensities, yet humans immediately recognize them as the same shape. Cosine distance captures that intuition surprisingly well.

What stood out most in the visualization was that the clusters remained well-defined without feeling artificially separated. Unlike t-SNE, where some clusters appeared isolated islands floating in space, UMAP preserved more of the broader organization of the dataset. Digits with similar visual characteristics often occupied nearby regions, and the overall arrangement felt more coherent.

The decision to use three dimensions also revealed something I would have missed in a standard 2D plot. Some groups that appeared partially overlapping in two dimensions unfolded more naturally when given an additional degree of freedom. The manifold had more room to express its structure, making the relationships between digits easier to interpret.

What I appreciated most about UMAP was that it felt less interested in creating the prettiest visualization and more interested in preserving a useful representation of the data. The clusters were slightly less dramatic than t-SNE, but they felt more trustworthy.

If PCA taught me that variance is not the same as separability, and t-SNE taught me to be cautious of beautiful plots, UMAP taught me that understanding data often requires balancing local detail with global structure. That balance is probably why UMAP has become the default visualization tool for many machine learning practitioners today.

The Surprisingly Interesting Sammon Mapping

Before this project, I had barely encountered Sammon Mapping.

Compared to PCA or t-SNE, it's rarely discussed.

After experimenting with it, I think that's unfortunate.

Sammon Mapping attempts to preserve pairwise distances during projection.

In other words, it's trying to stay faithful to the geometry of the original space.

The trade-off becomes obvious immediately.

It's computationally expensive.

The visualizations aren't as dramatic.

The clusters don't explode apart like they do with t-SNE.

But that's exactly the point.

Sammon Mapping is optimizing for honesty rather than visual appeal.

That made it one of the most interesting methods in the project.

When the KNN Graph Changed My Perspective

Most dimensionality reduction techniques represent data as points.

KNN Graphs represent data as relationships.

That sounds like a small difference.

It isn't.

Instead of asking:

Which cluster does this point belong to?

I found myself asking:

Which points connect different regions of the dataset?

The graph exposed bridge points, ambiguous digits, and unusual samples that were much harder to notice in traditional scatter plots.

It shifted my focus away from clusters and toward connectivity.

For exploratory analysis, that perspective can be incredibly valuable.

What Autoencoders Revealed

The Autoencoder was where the project started feeling less like classical machine learning and more like modern AI.

Unlike PCA or t-SNE, the Autoencoder isn't applying a predefined projection rule.

It's learning a representation.

The network compresses the input into a latent space and then attempts to reconstruct the original image.

To succeed, it must learn which information matters.

The resulting latent space felt fundamentally different from the classical methods.

It wasn't simply a compressed version of the original data.

It was a learned representation of the data.

The structure felt smooth.

Continuous.

Almost as though the digits existed on an underlying manifold rather than as isolated clusters.

For the first time, I could see why latent representations became such an important idea in deep learning.

The Lesson I Didn't Expect

I started this journey because PCA didn't improve a model.

I finished it with a completely different appreciation for exploratory data analysis.

The most important insight wasn't that one dimensionality reduction technique is better than another.

It was that every technique reveals a different aspect of the data.

PCA reveals variance.

LDA reveals separability.

t-SNE reveals neighborhoods.

UMAP balances local and global structure.

Sammon Mapping reveals geometry.

KNN Graphs reveal connectivity.

Autoencoders reveal learned representations.

The algorithms weren't competing.

They were answering different questions.

And that's probably the biggest lesson I took away from the project.

Before spending weeks tuning hyperparameters or experimenting with new models, it's worth asking a simpler question:

Do I actually understand the shape of my data?

Sometimes the fastest way to improve a model isn't another optimization trick.

It's developing a better intuition for the space your data lives in.

Explore the Project

I open-sourced the entire project so anyone can experiment with the visualizations themselves.

GitHub:
https://github.com/siva-rgb/Dim_Reduction

If you run it, I'd encourage you to spend less time looking for the "best" dimensionality reduction technique and more time asking:

What is each technique trying to tell me about the data?

That's where the interesting insights usually begin.