DEV Community: jacobjerryarackal

PCA (Principal Component Analysis): Finding the Hidden Structure in High‑Dimensional Data

jacobjerryarackal — Mon, 18 May 2026 17:57:54 +0000

1. The Problem It Solves

Imagine you have a dataset with 100 features, for example, pixel values from 10×10 grayscale images, or 100 different sensor readings from a factory machine. Visualising the data is impossible, many features might be redundant (e.g., two sensors that always move together), and training a model on all 100 features can be slow and prone to overfitting. PCA solves the dimensionality reduction problem: it finds a small number of new features (called principal components) that capture the most important patterns in the data. Real‑world applications include compressing images, visualising high‑dimensional data in 2D/3D, speeding up other algorithms, and removing noise.

2. The Core Idea (Intuition First)

Think of a flat, elongated cloud of points on a piece of paper. If you had to draw a single line through that cloud that “explains” as much of the shape as possible, you would draw the line along the longest direction of the cloud that’s the first principal component. Then, if you had to draw a second line, perpendicular to the first, that explains the next most variation, that’s the second principal component. PCA finds these “lines of maximum variance” in your data. By keeping only the top few such directions, you can represent each data point with just two or three numbers instead of hundreds, while losing as little information as possible.

Technically, PCA looks for orthogonal directions (principal components) that maximise the variance of the projected data. It turns out these directions are the eigenvectors of the data’s covariance matrix, sorted by their eigenvalues.

3. How It Works (The Math + Logic)

Let’s walk through PCA step by step.

Step 1 – Standardise the data

PCA is sensitive to scale. We first centre (subtract mean) and often scale (divide by standard deviation) each feature.

For a data matrix $X$ of shape $(nsamples,nfeatures)(n_{\text{samples}}, n_{\text{features}})$ , we compute the mean of each feature and subtract it:

X_{\text{centered}} = X - \bar{X}

Step 2 – Compute the covariance matrix

The covariance matrix $C$ (size $nfeatures×nfeaturesn_{\text{features}} \times n_{\text{features}}$ ) tells us how each pair of features varies together. Entry $C_{ij}$ is the covariance between feature $i$ and feature $j$ .

\mathbf{C} = \frac{1}{n-1} X_{\text{centered}}^T X_{\text{centered}}

Step 3 – Compute eigenvalues and eigenvectors

We find eigenvectors $v\mathbf{v}$ and eigenvalues $λ\lambda$ that satisfy:

\mathbf{C} \mathbf{v} = \lambda \mathbf{v}

Each eigenvector v is a principal component (a direction in the original feature space).
Its corresponding eigenvalue λ is the amount of variance explained by that component.

We sort the eigenvectors by decreasing eigenvalues. The first eigenvector points in the direction of highest variance, the second is orthogonal and points in the next highest, and so on.

Step 4 – Choose the top k components

We keep only the first k eigenvectors. The value k is often chosen by looking at the “explained variance ratio”: sum of the top k eigenvalues divided by the sum of all eigenvalues. A common rule is to keep enough components to explain 95% of the total variance.

Step 5 – Project the data

Finally, we transform the original data into the new lower‑dimensional space:

X_{\text{reduced}} = X_{\text{centered}} \cdot W_k

where $W_k$ is a matrix whose columns are the top $k$ eigenvectors. The new data has shape $(nsamples,k)(n_{\text{samples}}, k)$ .

4. When to Use It

Use PCA when:

You need to visualise high‑dimensional data in 2D or 3D (e.g., scatter plot of customers after PCA).
You want to speed up another algorithm (e.g., reduce 1000 features to 50 before training a k‑NN or SVM).
You suspect multicollinearity – many features are highly correlated. PCA decorrelates them.
You want to remove noise – small variance components often capture noise.

Assumptions:

PCA is linear, it only finds linear combinations of original features. If the underlying structure is highly non‑linear, autoencoders or t‑SNE may work better.
It assumes that directions of maximum variance are “important” for your downstream task. This is often true but not guaranteed (e.g., the smallest variance direction might separate classes better).

When it fails:

When features are not linearly related to the underlying structure (e.g., a spiral or circle pattern).
When you need interpretability, the principal components are linear mixes of all original features, so they are not as interpretable as the original features themselves.

5. My Implementation

We’ll use the classic Iris dataset (4 features) and reduce it to 2D for visualisation. Then we’ll see how much variance is preserved.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Load data
iris = load_iris()
X, y = iris.data, iris.target
feature_names = iris.feature_names
target_names = iris.target_names

# Standardise (important for PCA)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply PCA
pca = PCA(n_components=2)  # reduce to 2 dimensions
X_pca = pca.fit_transform(X_scaled)

# Explained variance ratio
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance preserved: {sum(pca.explained_variance_ratio_):.2%}")
print(f"Principal components (directions in original feature space):")
for i, comp in enumerate(pca.components_):
    print(f"  PC{i+1}: {', '.join([f'{name}: {val:.2f}' for name, val in zip(feature_names, comp)])}")

# Visualise
plt.figure(figsize=(8,6))
colors = ['red', 'green', 'blue']
for target, color, label in zip(np.unique(y), colors, target_names):
    plt.scatter(X_pca[y == target, 0], X_pca[y == target, 1], 
                c=color, label=label, alpha=0.7)
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')
plt.title('Iris dataset after PCA (2D projection)')
plt.legend()
plt.grid(alpha=0.3)
plt.show()

Output (numerical values may vary slightly):

Explained variance ratio: [0.72962445 0.22850762]
Total variance preserved: 95.81%
Principal components (directions in original feature space):
  PC1: sepal length (cm): 0.52, sepal width (cm): -0.27, petal length (cm): 0.58, petal width (cm): 0.56
  PC2: sepal length (cm): 0.38, sepal width (cm): 0.92, petal length (cm): 0.02, petal width (cm): 0.07

The first two principal components capture 95.8% of the total variance – meaning we lose very little information by going from 4 dimensions down to 2. The scatter plot clearly separates the three species, showing PCA preserved the meaningful structure.

6. Key Takeaways

PCA is a linear dimensionality reduction technique that finds directions of maximum variance using eigenvectors of the covariance matrix. It is fast, deterministic, and works well for many real‑world datasets.
Always standardise your data before PCA otherwise features with large scales will dominate the principal components, and the results will be misleading.
Use the explained variance ratio to decide how many components to keep. A common choice is to keep enough to explain 90–95% of the variance. For visualisation, use 2 or 3 components and accept some information loss.

XGBoost: When Gradient Boosting Meets Regularization

jacobjerryarackal — Fri, 15 May 2026 17:25:10 +0000

1. The Problem It Solves

Imagine you’re a loan officer at a bank. You have thousands of past loan applications with features like income, credit score, employment length, and debt-to-income ratio. You need to predict whether a new applicant will default or repay. This is a binary classification problem, but real-world data is messy: missing values, outliers, non-linear relationships, and interactions between features. Many algorithms struggle to handle all of this gracefully without heavy preprocessing. XGBoost (eXtreme Gradient Boosting) was built specifically to solve such tabular prediction problems with high accuracy, speed, and robustness. It’s become the go‑to algorithm for Kaggle competitions and many industry applications, from fraud detection to customer churn prediction.

2. The Core Idea (Intuition First)

Think of a group of friends trying to guess the weight of a cake. The first friend makes a rough guess say, 2 kg. The second friend doesn’t start from scratch; instead, she tries to correct the error of the first guess. If the real weight is 2.5 kg, the error is +0.5 kg, so she predicts +0.5 kg. The third friend corrects the remaining error, and so on. By combining many weak guesses (each slightly better than random), they arrive at a very accurate final estimate.

XGBoost works exactly like that: it builds an ensemble of decision trees sequentially. Each new tree tries to correct the mistakes made by all previous trees combined. But there’s a twist – XGBoost adds regularization to prevent overfitting, and it optimises the whole process to be lightning fast. It’s not just “gradient boosting” – it’s gradient boosting on steroids.

Technically, XGBoost minimises a regularised objective function that balances prediction error (loss) with model complexity. It uses a second‑order Taylor approximation of the loss (like Newton’s method) to guide tree splitting, which is more accurate than the simple gradient used in standard gradient boosting.

3. How It Works (The Math + Logic)

XGBoost builds an ensemble of $K$ decision trees. For a given prediction $y^i\hat{y}_i$ , it sums the outputs of all trees:

y^i=∑k=1Kfk(xi),fk∈F \hat{y}i = \sum{k=1}^{K} f_k(x_i), \quad f_k \in \mathcal{F}

where each $f_k$ is a tree (a mapping from features to leaf weights). The algorithm learns the trees one by one to minimise the following objective:

L(t)=∑i=1nℓ(yi,y^i(t−1)+ft(xi))+Ω(ft) \mathcal{L}^{(t)} = \sum_{i=1}^{n} \ell\bigl(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)\bigr) + \Omega(f_t)

$ℓ\ell$ is a differentiable loss function (e.g., log loss for classification, squared error for regression).
$y^i(t−1)\hat{y}_i^{(t-1)}$ is the prediction from the previous $t - 1$ trees.
$f_t$ is the new tree we are adding at step $t$ .
$Ω(f)=γT+12λ∑j=1Twj2\Omega(f) = \gamma T + \frac{1}{2} \lambda \sum_{j=1}^{T} w_j^2$ is the regularisation term: $T$ = number of leaves in the tree, $w_j$ = weight (prediction) on leaf $j$ , $γ\gamma$ and $λ\lambda$ are hyperparameters. This penalises complex trees (many leaves or large leaf weights), reducing overfitting.

XGBoost uses a second‑order approximation of the loss (Newton's method) to make optimisation efficient. For a given tree structure, the optimal leaf weight and the resulting gain from a split are derived analytically. When deciding where to split a node, XGBoost tries every feature and every possible split value, computing the "gain":

\text{Gain} = \frac{1}{2} \left[ \frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{(G_L+G_R)^2}{H_L+H_R + \lambda} \right] - \gamma

Here $G$ = sum of first derivatives (gradients) in a leaf, $H$ = sum of second derivatives (Hessians). A split is made only if the gain exceeds $γ\gamma$ , which directly prunes leaves.

The algorithm also includes:

Column subsampling (like Random Forest) – reduces overfitting and speeds up training.
Handling missing values – learns the best direction to send missing values.
Weighted quantile sketches – efficiently finds approximate split points for large datasets.

After building $K$ trees, you have a powerful, regularised ensemble.

4. When to Use It

Best for:

Medium‑sized to large tabular datasets (thousands to millions of rows, dozens to hundreds of features).
Problems where you need high accuracy without extensive feature engineering – XGBoost can learn non‑linear interactions and handle mixed data types (numeric + categorical, though categorical needs encoding).
Situations where interpretability is secondary to performance (you can get feature importance, but a single tree is easier to explain).

Assumptions:

XGBoost makes no strong assumptions about data distribution. It works well even if features are correlated or if there are irrelevant features (thanks to regularisation).

When it fails:

Very high‑dimensional sparse data (like text or image pixels) – deep learning usually works better.
Small datasets (less than a few hundred rows) – simple models like logistic regression or a single decision tree often outperform and are less prone to overfitting.
Real‑time latency‑critical applications – XGBoost prediction is fast, but an ensemble of 100 trees is slower than a linear model. For microsecond latency, consider simpler models or use specialised hardware.
Non‑tabular data (images, sequences, graphs) – use CNNs, RNNs, or Graph Neural Networks instead.

My opinion: XGBoost is my first choice for any supervised learning problem on structured data. I’ve seen it beat carefully tuned neural networks on multiple Kaggle competitions. The only reason to not use it is when you desperately need interpretability (then use logistic regression or a single decision tree) or when you have a tiny dataset.

5. Implementation

Below is a complete example using XGBoost for classification on the famous breast cancer dataset. We’ll train a model, evaluate it, and show feature importance.

import xgboost as xgb
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

# Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Create XGBoost classifier
model = xgb.XGBClassifier(
    n_estimators=100,        # number of trees
    max_depth=6,             # maximum tree depth
    learning_rate=0.1,       # step size shrinkage
    subsample=0.8,           # row subsampling
    colsample_bytree=0.8,    # column subsampling per tree
    reg_lambda=1.0,          # L2 regularisation on leaf weights
    reg_alpha=0.0,           # L1 regularisation (optional)
    random_state=42,
    eval_metric='logloss'
)

# Train
model.fit(X_train, y_train)

# Predict
y_pred = model.predict(X_test)

# Evaluate
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=data.target_names))

# Feature importance
importance = model.feature_importances_
top_indices = np.argsort(importance)[-5:]   # top 5 features
print("\nTop 5 most important features:")
for idx in top_indices[::-1]:
    print(f"  {data.feature_names[idx]}: {importance[idx]:.3f}")

Output:

Accuracy: 0.9737

Classification Report:
              precision    recall  f1-score   support
   malignant       0.97      0.97      0.97        42
      benign       0.98      0.98      0.98        72

    accuracy                           0.97       114
   macro avg       0.97      0.97      0.97       114
weighted avg       0.97      0.97      0.97       114

Top 5 most important features:
  worst concave points: 0.152
  worst perimeter: 0.121
  worst texture: 0.089
  mean concave points: 0.074
  worst area: 0.068

The model achieves ~97% accuracy on the test set with almost no tuning – that’s the power of XGBoost. You can see which features drove the decision (concave points and perimeter are highly predictive for breast cancer).

6. Key Takeaways

XGBoost is gradient boosting with regularisation and second‑order optimisation – it’s faster, more accurate, and less prone to overfitting than plain gradient boosting. Always try it as a baseline for tabular data.
It handles real‑world messiness well – missing values, outliers, non‑linear relationships, and feature interactions are all taken care of internally, saving you hours of preprocessing.
Hyperparameter tuning matters – start with n_estimators=100, max_depth=6, learning_rate=0.1, then use subsample and colsample_bytree to reduce overfitting. For large datasets, enable the GPU (tree_method='gpu_hist') for massive speedups.

Why “Just Prompting” Fails on Private Data: A RAG Post‑Mortem

jacobjerryarackal — Thu, 14 May 2026 08:04:27 +0000

The Problem

You have a 400‑page internal handbook includes compliance rules, HR policies, engineering runbooks. You ask an LLM: “What’s the approval chain for a budget over $50k?”

Without RAG, the model hallucinates: “The VP of Finance and the CTO must both approve.” But your real policy says: “Only the CFO for >$50k, plus a board note if >$200k.”

The core problem: LLMs are frozen at training time. They don’t know your private documents. Fine‑tuning is expensive, lags behind updates, and still suffers from parametric knowledge bleed. RAG solves the specific problem of grounding generation in fresh, proprietary, or long‑tail facts without retraining.

But naïve RAG (chunk → embed → retrieve → stuff into prompt) breaks in surprising ways. This article walks through one real failure, three common failure modes, and the guardrails we built to make RAG production‑ready.

The Dry‑Run: Answering an Employee’s Parental Leave Question

Scenario: An employee asks a Slack bot: “How many weeks of paid parental leave do I get, and do I need to notify HR before birth?”

The source is a 50‑page PDF Parental Leave Policy v4.2, last updated 3 months ago.

Step 1 – Chunking

We split the PDF into overlapping chunks of 512 tokens (with 128‑token overlap).

Why? Without overlap, a sentence like “The leave period is 12 weeks. However, for birth mothers, an additional 4 weeks of medical recovery applies.” might split right after “12 weeks”, losing the exception.

Step 2 – Embedding & Indexing

Each chunk is passed through text-embedding-3-small (1536 dimensions). We store vectors in a pgvector index together with metadata (page number, section title, last update date).

Step 3 – Query Embedding

User query: “paid parental leave weeks + HR notification before birth?”

We embed the query. Note: we deliberately do not use a separate rewriter; the raw query goes to the retriever.

Step 4 – Retrieval

Vector similarity (cosine) returns top‑5 chunks. Example chunks retrieved:

“Eligible employees receive 12 weeks of fully paid parental leave.” (score 0.92)
“Birth mothers may take an additional 4 weeks of paid medical recovery leave, distinct from parental leave.” (score 0.89)
“Notification: Employee must submit a leave request in Workday at least 30 days before the expected birth date.” (score 0.87)
“Adoptive parents receive the same 12 weeks but no medical recovery weeks.” (score 0.76)
“Leave can be taken intermittently with manager approval.” (score 0.68)

Step 5 – Generation Prompt

We assemble a prompt:

You are an HR assistant. Use ONLY the following context to answer the question.
If the answer is not in the context, say "I don't know."

Context:
[chunk1] [chunk2] [chunk3]

Question: How many weeks of paid parental leave do I get, and do I need to notify HR before birth?

Answer in a clear, bulleted list.

Step 6 – LLM Response

The model correctly outputs:

12 weeks of fully paid parental leave for all eligible employees.
Birth mothers get an additional 4 weeks of paid medical recovery leave.
You must notify HR via Workday at least 30 days before the expected birth date.

Success – no hallucination about a “CTO approval”.

Failure Modes (Where RAG Secretly Fails)

Even with the above, we see three catastrophic failure patterns in production.

Failure 1 – The “Lost in the Middle” Problem

Our top‑5 chunks are concatenated. The LLM pays attention to the first and last chunks, but the middle ones (e.g., the notification rule) are ignored.

Consequence: The bot answers the weeks question but omits the 30‑day notification rule. Employee misses the deadline.

Failure 2 – Low‑Relevance Retrieval (But High Cosine Score)

A query like “What happens if I return to work part‑time after leave?”

Embedding returns a chunk: “Intermittent leave requires manager approval” (cosine 0.81), but the actual policy says “Returning part‑time is not allowed during the first 12 weeks.” That chunk exists but has low embedding similarity (0.52) because it uses different wording (“reduced schedule” vs “part‑time”).

Consequence: The model says “manager can approve” – wrong and harmful.

Failure 3 – Contradictory Chunks

Two chunks in the same document:

Chunk A: “You may use PTO during parental leave to top up pay.” (old version)
Chunk B: “As of Jan 2025, PTO cannot be used to top up parental leave pay.”

The retriever returns both. The LLM picks one at random, or hallucinates a compromise.

Consequence: Inconsistent answers depending on chunk order.

Guardrails (Engineering Fixes for Each Failure)

We implemented five explicit guardrails on top of the basic RAG pipeline.

Guardrail 1 – Reranking with Cross‑Encoder

After vector retrieval, we take top‑20 chunks and rerank using a cross‑encoder (cross-encoder/ms-marco-MiniLM-L-6-v2). This model directly computes relevance of (query, chunk) pairs.

Result: The “part‑time return” chunk scores 0.92 after reranking, while “intermittent leave” drops to 0.43. We keep only top‑3 reranked chunks.

Guardrail 2 – Chunk Positioning Weighting

In the prompt, we present chunks as numbered sources. We append a sentence: “The middle sources are often the most detailed – do not skip them.”

We also use a metadata field chunk_position_in_document and instruct the LLM to cite at least two different positions.

Guardrail 3 – Contradiction Detector

Before sending chunks to the LLM, we run a lightweight entailment model (roberta-large-mnli) to check for contradictions. If two chunks have CONTRADICTION score > 0.8, we include both but add a system instruction: “The following two sources contradict each other. Explain the discrepancy and default to the newer one based on document version.”

Guardrail 4 – Forced Citation

We require the LLM to output citations like [src: page 12]. We parse the response. If any statement lacks a citation, we reject and retry with a stricter prompt.

Guardrail 5 – Hybrid Search

We augment vector search with BM25 keyword matching. For queries with rare terms (e.g., “Workday notification”), BM25 finds the exact phrase that embedding might smooth over. Final score = 0.6 * vector + 0.4 * BM25.

Architecture Diagram

Explanation of the diagram:

User query is processed in parallel (embedding + BM25).
Vector + BM25 results are fused (not shown for simplicity, but it’s inside the DB step).
Reranker reduces to top‑5 most relevant chunks.
Contradiction detector adds metadata before prompting.
LLM generates, then citation validator enforces groundedness. Retry loop prevents hallucinated claims.

Conclusion

RAG is not “just glue code”. Without reranking, contradiction detection, and forced citations, your bot will confidently produce wrong answers from the same document. The guardrails above have reduced hallucination rate on our internal HR dataset from 23% to 4.7% (measured by human‑evaluated citation correctness).

The Copy‑Paste Trap: When Knowing “Everything” Still Leaves You with Nothing

jacobjerryarackal — Fri, 24 Apr 2026 14:21:26 +0000

I’m a full‑stack MERN developer. I work with Next.js, PostgreSQL, GraphQL, Docker, Python, RAG pipelines, and Generative AI. I can spin up a production app in a weekend, wire a vector database into a chat interface, and deploy a serverless function before my coffee gets cold.

Last month I integrated a RAG system for a client. Chat, Grok, Claude, DeepSeek‑something gave me a beautiful, 200‑line chunking-and-embedding function. It worked perfectly on the first run. I felt unstoppable. A week later the client asked a simple question: “Why did you choose that embedding model over the cheaper one?” I froze. I had no idea. I hadn’t made a choice – I’d just copied the answer.

That’s when I saw the trap.

We’ve mistaken copy‑paste productivity for genuine understanding, and our AI assistants are more than happy to play along.

The illusion no one talks about

AI‑assisted coding gives you a warm, dopamine‑rich feedback loop: paste a prompt, get working code, ship fast, feel like a 10x engineer. I know the entire stack Next.js, Postgres, GraphQL, Docker, Python but somewhere along the way I stopped exercising that knowledge. I outsourced my thinking to a black box that doesn’t care if I learn.

The result? Systems that run but can’t be justified. Pull requests that pass but can’t be explained. A developer who is simultaneously “senior” on paper and helpless when the prompts stop giving perfect answers.

Five symptoms that hit close to home

You ship a feature in an hour and can’t explain it tomorrow.

If I asked you to whiteboard the data flow of that GraphQL mutation you dropped in yesterday without looking at the code, could you?
Your Docker Compose file works because the AI said so.

You don’t know why that depends_on with a healthcheck is there, or what happens if you remove the init: true flag.
You avoid reading source code because the LLM summarizes it for you.

Why dig into the pg‑promise docs or the Next.js middleware logic when a chatbot can just tell you what to type?
You feel a spike of anxiety when the AI goes down.

If your internet cut out right now, could you still write a production‑ready API route from memory?
Your commits are full of code you’d never write yourself.

Slick one‑liners, clever.reduce() chains, async patterns you don’t fully trust – but hey, the tests passed.

How we got here (it’s not entirely your fault)

The pressure to deliver makes this illusion irresistible. Managers celebrate velocity, not understanding. Imposter syndrome whispers that if you don’t ship fast, someone with better prompt engineering will replace you. And the tools are genuinely amazing – they make us feel like magicians. But magic isn’t engineering.

The real stack isn’t just tools, it’s ownership

Real understanding in a MERN + AI world is not about memorizing syntax. It’s about holding a mental model so clear that you can:

Diagnose a slow GraphQL resolver by reasoning through the query planner, not by pasting the slow query into ChatGPT.
Decide whether to use getServerSideProps or incremental static regeneration by understanding the trade‑offs, not because the AI recommended one.
Build a RAG pipeline where you chose the chunk size based on your data’s semantic shape, not just because “the tutorial used 512 tokens.”

The uncomfortable truth: Copy‑paste mastery is still just copy‑paste.

Breaking the spell: uncomfortable practices that work

I don’t want to abandon AI. I want to use it as a teacher, not a crutch. Here’s what I’m doing to rebuild deep understanding:

1. Close the tab and build it from scratch

Pick a feature you’ve copied recently a Next.js API route, a Dockerfile, a Python ingestion script. Delete the AI‑generated code. Now rebuild it with only the official docs (and your brain). This hurts. It’s supposed to.

2. Break it deliberately

Take that working Docker Compose file. Remove a service dependency. Before running docker compose up, predict the exact error. Did you guess right? If not, the AI still owns that piece of your system.

3. Teach the brick

Explain your GraphQL schema to an imaginary junior developer in under three minutes. Use no jargon. If you stumble on “why we used a dataloader here,” you just found a gap.

4. Trace one request through the entire stack

In a Next.js app with GraphQL and Postgres, follow a single user login from the browser click, through the middleware, into the resolver, down to the SQL query, and back. Draw it on paper. You’ll be shocked how much is fuzzy.

5. Ask “why” five times and refuse AI answers

“Why did we use PostgreSQL instead of MongoDB for this feature?” The first answer is easy. The fifth answer (where you hit the limits of your knowledge) is where growth lives. Do not let an LLM shortcut that loop.

What’s on the other side

When you build real understanding, the anxiety vanishes. A production bug stops being a panic attack and becomes a puzzle you’re equipped to solve. You can justify every architectural decision to a client or a CTO. You stop being the person who “knows the whole stack” on paper, and start being the person who actually owns it debugger, profiler, architect, and all.

Your challenge this week

Pick one piece of code you shipped in the last month that came almost entirely from an AI assistant. It can be a Next.js page, a Docker setup, a Python RAG utility, anything. Now ask yourself: If I had to rebuild this without any AI, could I? If the honest answer is “no,” you just found your most important learning task.

Don’t fix it with another prompt. Fix it by sitting with the discomfort, opening the docs, and building real mastery. The AI will still be there when you’re done only this time, you’ll be the one using it, not the other way around.

I Built a RAG Pipeline. Then I Realized Retrieval Is the Real Model

jacobjerryarackal — Wed, 08 Apr 2026 03:03:07 +0000

Everyone talks about the LLM. GPT‑4, Claude, Gemini – that’s the celebrity. But after building my first real RAG pipeline, I learned something humbling: the LLM is the interchangeable part. The retrieval system is the actual worker.

Let me show you what I mean.

The 4‑Step Pipeline We All Copy

You’ve seen the tutorial code a hundred times:

Ingest – chunk your documents
Embed – turn chunks into vectors
Retrieve – find top‑k similar chunks
Generate – LLM answers with that context

It works. My bot could answer company policy questions with citations. I felt smart.

Then I asked: “Can I get a refund for a digital product?”

The LLM gave a beautiful, confident answer which was completely wrong. Because my retrieval returned a chunk about physical returns (30 days, original packaging) and completely missed the digital product exception sitting two paragraphs away.

The LLM did its job perfectly. The retrieval failed.

Why Retrieval Is the Real Model

Here’s what I learned the hard way:

What you think matters	What actually matters
Which LLM you use	How you chunk documents
Prompt engineering	Embedding quality
System prompts	Re‑ranking after retrieval

The LLM just formats the answer. Retrieval decides whether the answer is true.

The Code That Fixed My Pipeline

Semantic search alone misses exact phrases like “non‑refundable after download”. Keyword search alone misses meaning. Hybrid search combines both. Here’s the core (using FAISS + BM25):

from sentence_transformers import SentenceTransformer
import faiss, numpy as np
from rank_bm25 import BM25Okapi

# 1. Load documents and embed
docs = ["Refund within 30 days, physical items only.",
        "Digital products: non-refundable after download.",
        "Contact support for defective digital items."]
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(docs)
index = faiss.IndexFlatL2(embeddings.shape[1])
index.add(np.array(embeddings, dtype='float32'))

# 2. BM25 keyword index (tokenized)
tokenized_docs = [doc.lower().split() for doc in docs]
bm25 = BM25Okapi(tokenized_docs)

# 3. Hybrid search function
def hybrid_search(query, top_k=2, alpha=0.5):
    # Semantic score (distance -> similarity)
    query_vec = model.encode([query])
    distances, indices = index.search(query_vec, top_k)
    semantic_scores = 1 / (1 + distances[0])  

    # Keyword score
    query_tokens = query.lower().split()
    bm25_scores = bm25.get_scores(query_tokens)
    top_bm25_idx = np.argsort(bm25_scores)[-top_k:][::-1]
    keyword_scores = [bm25_scores[i] for i in top_bm25_idx]

    # Combine (normalized)
    combined = {}
    for i, idx in enumerate(indices[0]):
        combined[idx] = alpha * semantic_scores[i]
    for i, idx in enumerate(top_bm25_idx):
        combined[idx] = combined.get(idx, 0) + (1-alpha) * (keyword_scores[i] / max(keyword_scores))

    return sorted(combined.items(), key=lambda x: x[1], reverse=True)[:top_k]

# 4. Test
query = "Can I get my money back for a digital product?"
results = hybrid_search(query)
for idx, score in results:
    print(f"Score: {score:.2f} | {docs[idx]}")
# Output: Score: 0.92 | Digital products: non-refundable after download.

That alpha=0.5 balances meaning and exact wording. Without hybrid search, the digital product chunk was #3 (ignored). With it, #1.

Three Changes That 10x’ed My Pipeline

Chunk size is not a default – Moved to overlapping chunks (200 tokens with 50 overlap).
Semantic search alone lies – Added BM25 hybrid search (see code above).
Re‑ranking changes everything – A small cross‑encoder re‑scored top‑10 chunks, lifting accuracy from 72% to 91%.

The Mistake Most People Make

We treat RAG as an LLM problem. So we tweak prompts, swap models, add system instructions.

But the LLM is forced to use whatever context you give it. If you feed it the wrong chunk, it will hallucinate confidently. If you feed it the right chunk, even a small model answers correctly.

The bottleneck is almost never the LLM. It’s the retriever.

What I Do Differently Now

Before I write a single line of agent code, I ask three questions:

“If I searched my vector database by hand, would I find the exact sentence that answers this?”
“Does my retrieval work for synonyms AND exact keywords?” → if no, hybrid search.
“Is the top‑1 retrieved chunk actually the best?” → if no, add a re‑ranker.

The Bottom Line

The AI industry sells you on the model. But in production RAG systems, the model is the cheapest, most replaceable component. The hard part – the part that separates working bots from demoware – is getting the right information into the context window.

The LLM is the pen. Retrieval is the memory. And memory is what makes a system useful.

So next time your RAG bot fails, don’t blame GPT. Look at what you retrieved. I promise that’s where the real problem lives.

We Let an LLM Control a File System and Run Commands – Here’s What Actually Broke First

jacobjerryarackal — Sat, 04 Apr 2026 08:02:02 +0000

I wanted to push an LLM beyond simple chat and see if it could actually build real code.

So I gave it direct access to the file system and the ability to run terminal commands. The task was straightforward: “Create a clean React login page with email, password, remember-me checkbox, and form validation.”

It started confidently. Within minutes everything broke.
The System We Built
We connected two tools to the LLM:

file_system (list, read, write, delete files)
run_command (execute npm, start dev server, etc.)

We used MCP (the “USB-C for AI” protocol) so the model could call tools cleanly. The goal was to let the LLM act like a real developer — explore the folder, create files, install packages, and test the app.
It sounded simple. It was not.

Failure #1: It Assumed the Project Already Existed
What broke:
The model immediately started writing Login.jsx in an empty folder. No package.json, no React setup, no dependencies.
Why it broke:
The LLM had no understanding of project bootstrapping. It assumed a full React app was already there.
What we learned:
We had to explicitly tell it “first create the project structure” in every new session. This became our first mandatory step.

Failure #2: It Ran Commands at the Wrong Time
What broke:
After creating a few files, it ran npm start and npm run build before any dependencies were installed. The terminal exploded with 47 errors.
Why it broke:
The model treated commands like a checklist instead of understanding dependencies. It didn’t realise you can’t run the app before npm install.
What we learned:
We added a rule: never run npm start or npm run build until package.json exists and all dependencies are installed. This single rule saved us from multiple crashes.

Failure #3: It Mixed Concerns and Created Messy Code
What broke:
It put all the Tailwind CSS and form logic inside a single Login.jsx file. The component became 180 lines long, impossible to read, and had styling mixed with business logic.
Why it broke:
The model was optimising for “one file = done” instead of proper component structure.
What we learned:
We had to force it to create separate files (Login.jsx, Login.css, utils/validation.js). Once we added this constraint, the code quality jumped dramatically.

Failure #4: It Had No Memory of Previous Mistakes
What broke:
Even after we fixed the directory issue, in the next loop it tried to create the same wrong file again in the wrong location.
Why it broke:
The model had no persistent memory of what it had already tried and failed.
What we learned:
We started saving a small agent-log.md file after every loop so the model could read its own history before making the next decision. This simple trick reduced repeated mistakes by almost 70%.
After 8 loops and 14 minutes, we finally had a clean, working React login page with proper validation and structure.

The Real Lesson
The LLM wasn’t the problem. The problem was that we treated it like a magician instead of a junior developer with superpowers.
Once we gave it real tools (file system + terminal) and forced it to work inside real constraints, it went from completely broken to actually useful.

In 2026, the biggest unlock isn’t a smarter model.
It’s giving the model the right tools and the right guardrails.
I no longer ask LLMs to “write me some code.”
I give them a file system, terminal access, and clear rules.
That single change is what turns toys into tools you can actually ship.

“Prompt Engineering Is Enough” Is Wrong – Here’s What I Had to Add

jacobjerryarackal — Thu, 02 Apr 2026 12:10:36 +0000

I used to believe the hype.

I thought if I just wrote better prompts clearer instructions, few‑shot examples, chain‑of‑thought so that I could make any LLM do whatever I wanted. I spent weeks refining prompts, tweaking wording, adding “think step by step” like it was magic.

Then I tried to build something useful.

I asked the model to check the weather and tell me if I needed an umbrella. The response was confident and completely wrong. It hallucinated the forecast based on its training cut‑off. No real data, just made‑up facts wrapped in perfect English.

The prompt was excellent. The model was powerful. The result was useless.

That’s when I realised the uncomfortable truth: prompt engineering is not enough.

I went back and added one thing that actually fixed it tools.

I gave the model a simple weather tool and forced it to use a basic OBSERVE → THINK → ACT loop. Nothing fancy. Just three steps every single time.

Here’s what happened.

First loop – OBSERVE

The model receives my request: “Check the weather in Kochi and tell me if I need an umbrella today.” It sees it has access to a tool called get_weather(location).

Second loop – THINK

Instead of guessing, it reasons out loud: “I don’t have current weather data. I should use the tool to get real information.”

Third loop – ACT

It calls the tool in clean JSON:

{
  "tool": "get_weather",
  "arguments": {
    "location": "Kochi, Kerala"
  }
}

The tool returns actual data. The model goes back to THINK mode, sees the result (“light rain expected”), and only then generates the final answer: “Yes, take an umbrella.”

No hallucination. No made‑up weather. Just one tool + one loop.

I tested it on something harder, building a small React login page from scratch. With pure prompting, the model produced broken, outdated code and confidently told me it was correct. After adding file_system and run_command tools plus the same OBSERVE‑THINK‑ACT loop, it actually listed the directory, read the existing package.json, wrote proper components, fixed its own bugs, and shipped working code across eight loops.

The model was the same. The prompts were similar. The only difference was that I stopped treating the LLM as a magic oracle and started treating it as a brain that needs hands.

I also added MCP, the simple protocol everyone now calls “USB‑C for AI.” It made plugging in new tools ridiculously easy. No custom glue code. Just declare the tool once, and the agent knows exactly how to call it.

The change was night and day.

I stopped wasting time writing longer and longer prompts. I started adding tools and a reliable loop instead. The results went from “impressively wrong” to “actually useful.”

The lesson for 2026 is brutally simple: prompt engineering is table stakes, not the complete solution. If your LLM keeps hallucinating, forgetting tasks, or failing at real work, stop tweaking the prompt. Give it proper tools and a structured loop to use them.

The model is the brain. Tools are the hands. Without hands, even the smartest brain is stuck guessing.

I no longer believe “prompt engineering is enough.”

I now know exactly what I have to add.