DEV Community: Vineet Chauhan

What Really Happens When You Search on Netflix, Spotify, or Amazon? Behind the Scenes of Modern Search Systems

Vineet Chauhan — Fri, 03 Jul 2026 14:49:12 +0000

We type a few words. Millions of computations happen.

Every day, billions of people search for something.

"Interstellar" on Netflix
"Shape of You" on Spotify
"Wireless Mouse" on Amazon

The results appear almost instantly.

To most users, it feels simple. But behind that tiny search box is one of the most sophisticated engineering systems ever built.

Search is no longer just about finding matching words. Modern platforms try to understand what you mean, predict what you're looking for, personalize the results, rank millions of possibilities, and deliver everything within a fraction of a second.

In this article, we'll follow the complete journey of a search query—from the moment you press Enter to the instant the perfect result appears on your screen.

Step 1: You Type a Query

Imagine you open Amazon and type:

wireless gaming mouse

Your computer sends this request to Amazon's servers.

User
   │
   ▼
Search Box
   │
   ▼
Amazon Search Server

At this point, the system doesn't simply compare text with product names.

Instead, it starts understanding the query itself.

Step 2: Understanding the Query

Humans naturally understand language.

Computers don't.

So the first task is query preprocessing.

The search engine breaks your sentence into meaningful pieces.

Original Query

Wireless Gaming Mouse

After preprocessing:

wireless
gaming
mouse

Then several operations happen.

Convert to lowercase

Wireless → wireless

Remove unnecessary words

"The Best Wireless Mouse"

↓

best
wireless
mouse

Correct spelling

wirless

↓

wireless

Understand synonyms

TV

↓

Television

Shoes

↓

Sneakers

Amazon, Netflix and Spotify all maintain massive synonym dictionaries.

Step 3: Searching the Index

Suppose Amazon has 500 million products.

Would it compare your query against every product?

Absolutely not.

That would take far too long.

Instead, search engines build something called an Inverted Index.

Think of it like the index at the back of a textbook.

Instead of reading every page, you jump directly to the pages containing a keyword.

Example:

wireless

↓

Product 4
Product 18
Product 52
Product 190

gaming

↓

Product 18
Product 77
Product 190

mouse

↓

Product 18
Product 190
Product 250

The engine quickly finds products containing all three terms.

This reduces the search space from millions of products to only a few thousand.

Step 4: Keyword Matching Isn't Enough

Imagine searching for

Laptop Stand

A product named

Adjustable Notebook Holder

might never appear.

Even though both mean almost the same thing.

Traditional keyword search struggles here because the words are different.

Modern search engines solve this using semantic search.

Step 5: Understanding Meaning with Embeddings

AI converts every word, sentence or product into a list of numbers called an embedding.

Instead of storing words...

Dog

the AI stores something like

[0.42, -0.81, 1.55, ...]

Words with similar meanings appear close together inside this mathematical space.

Dog -------- Puppy

Cat -------- Kitten

Movie -------- Film

Laptop -------- Notebook

Now when someone searches

Notebook

the system can still recommend laptops because their embeddings are close together.

This is why modern search feels much smarter than searching a PDF with Ctrl + F.

Practical Example in Python

Using Sentence Transformers, we can perform semantic search in just a few lines.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer("all-MiniLM-L6-v2")

products = [
    "Wireless Gaming Mouse",
    "Mechanical Keyboard",
    "Gaming Laptop",
    "Bluetooth Speaker"
]

query = "Wireless Mouse"

product_embeddings = model.encode(products)
query_embedding = model.encode([query])

scores = cosine_similarity(query_embedding, product_embeddings)

for product, score in zip(products, scores[0]):
    print(product, round(score,3))

Instead of matching exact words, the model compares meaning.

Step 6: Ranking the Results

Suppose the search finds 5,000 matching products.

Should they all appear randomly?

Of course not.

The system now ranks them.

Some ranking signals include:

Relevance
Popularity
Ratings
Reviews
Purchase history
Availability
Delivery speed
User preferences
Click-through rate
Conversion rate

A simplified ranking formula looks like:

Final Score

=

0.4 × Relevance

+

0.3 × Popularity

+

0.2 × Rating

+

0.1 × Personalization

The highest-scoring products appear first.

Step 7: Personalization

Now comes one of the most powerful components.

The same search query can produce completely different results for different users.

Suppose two people search:

Shoes

User A

Previously searched:

Football
Nike
Adidas

Amazon is likely to recommend football boots.

User B

Previously searched:

Running
Marathon
Sports Watch

Amazon is more likely to recommend running shoes.

The query is identical.

The user is different.

So the results are different.

Step 8: Recommendation Systems

Search isn't the only way users discover content.

Recommendations are equally important.

Netflix recommends movies.

Spotify recommends songs.

Amazon recommends products.

These systems use several techniques:

Collaborative Filtering

People with similar interests often enjoy similar items.

If thousands of users who watched

Interstellar

also watched

Inception

Netflix learns this relationship.

Content-Based Filtering

Recommend similar items.

Action movie

↓

More action movies.

Rock song

↓

More rock songs.

Gaming laptop

↓

Gaming accessories.

Hybrid Recommendation

Most companies combine multiple recommendation methods with machine learning models.

This provides significantly better recommendations than relying on a single technique.

Step 9: Why Results Are Instant

Imagine rebuilding the ranking every time someone searches.

It would be far too slow.

Companies use multiple optimization techniques:

Caching
Distributed databases
Load balancing
Parallel processing
Search indexes
Vector databases
Content Delivery Networks (CDNs)

These systems reduce search latency from seconds to milliseconds.

Real Architecture

User

↓

Search API

↓

Query Processing

↓

Keyword Search

+

Semantic Search

↓

Ranking Engine

↓

Recommendation Engine

↓

Personalization Layer

↓

Final Results

Each block may itself contain dozens of microservices working together.

Why Vector Databases Matter

Traditional databases answer questions like:

Find all products costing less than ₹2000.

Vector databases answer questions like:

Find products that are semantically similar to this product.

Popular vector databases include:

FAISS
Pinecone
Milvus
Weaviate
ChromaDB
Qdrant

They power modern AI search and Retrieval-Augmented Generation (RAG) systems.

Where Machine Learning Fits In

Machine learning improves search in many ways:

Spell correction
Query understanding
Ranking
Recommendations
Fraud detection
Personalized search
Demand prediction

The goal isn't just to find a matching result—it's to predict the result you're most likely to choose.

Key Takeaways

Modern search systems are much more than databases with a search bar. They combine information retrieval, natural language processing, recommendation systems, machine learning, distributed computing, and efficient data structures to deliver highly relevant results in milliseconds.

The next time you search for a movie on Netflix, a song on Spotify, or a product on Amazon, remember that your request isn't simply matching text. It's being interpreted, expanded, ranked, personalized, and optimized through dozens of intelligent systems working together behind the scenes.

That tiny search box represents years of engineering innovation—and it's one of the best examples of how data structures, algorithms, AI, and software engineering come together to create an experience that feels almost effortless.

Gradient Descent: The Engine That Made Deep Learning Possible : How one simple idea changed the way machines learn

Vineet Chauhan — Fri, 03 Jul 2026 14:45:02 +0000

When I first started learning deep learning, I thought the magic was inside the model architecture.

CNNs looked powerful.

RNNs looked intelligent.

Transformers looked almost impossible to understand.

But slowly I realized something important.

Architecture is only one part of deep learning.

The real question is:

How does the model actually learn?

A neural network may contain millions or even billions of parameters.

But at the beginning, all those parameters are almost useless.

They are usually random numbers.

The model does not know anything.

It does not understand images.

It does not understand language.

It does not understand patterns.

So the real magic is not that a neural network has many parameters.

The real magic is that it can adjust those parameters automatically.

That automatic adjustment is made possible by Gradient Descent.

The Basic Problem

Suppose we are training a simple model.

y_pred = w * x + b

Here:

w = weight
b = bias

At first, the model makes wrong predictions.

So we calculate error.

loss = (y_actual - y_pred) ** 2

The goal is simple:

Reduce the loss

But the question is:

How should w and b change?

Should weight increase?

Should weight decrease?

By how much?

This is where Gradient Descent comes in.

What Gradient Descent Really Means

Gradient Descent simply means:

Move the parameters in the direction where loss decreases.

Imagine standing on a mountain in fog.

You cannot see the full path.

You only know the slope under your feet.

So you move downward step by step.

That is Gradient Descent.

In machine learning:

Mountain height = Loss
Position = Parameters
Downward direction = Negative gradient

The formula is:

new_weight = old_weight - learning_rate × gradient

or:

w = w - lr * dw
b = b - lr * db

This small formula is one of the biggest reasons deep learning works.

A Very Small Numerical Example

Suppose our model is:

y_pred = w * x

Let:

x = 2
y_actual = 10
w = 1

Prediction:

y_pred = 1 × 2 = 2

Loss:

loss = (10 - 2)² = 64

The prediction is too small.

So weight should increase.

Gradient Descent tells us exactly how much to update.

For squared error:

loss = (y - wx)²

Gradient with respect to weight:

dL/dw = -2x(y - wx)

Now substitute values:

dL/dw = -2 × 2 × (10 - 2)
      = -4 × 8
      = -32

Let learning rate be:

lr = 0.1

Update:

new_w = old_w - lr × gradient
new_w = 1 - 0.1 × (-32)
new_w = 1 + 3.2
new_w = 4.2

Now prediction becomes:

y_pred = 4.2 × 2 = 8.4

Earlier prediction was 2.

Now it is 8.4.

Much closer to 10.

That is learning.

Not memorization.

Not magic.

Just repeated improvement.

Code: Gradient Descent From Scratch

import numpy as np
import matplotlib.pyplot as plt

x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 6, 8, 10])

w = 0
b = 0

lr = 0.01
epochs = 1000

losses = []

for epoch in range(epochs):
    y_pred = w * x + b

    loss = np.mean((y - y_pred) ** 2)
    losses.append(loss)

    dw = (-2 / len(x)) * np.sum(x * (y - y_pred))
    db = (-2 / len(x)) * np.sum(y - y_pred)

    w = w - lr * dw
    b = b - lr * db

print("Final weight:", w)
print("Final bias:", b)

Expected output:

Final weight: close to 2
Final bias: close to 0

The model discovers:

y = 2x

by updating weights again and again.

Plotting The Loss Curve

plt.plot(losses)
plt.xlabel("Epochs")
plt.ylabel("Loss")
plt.title("Loss Decreasing During Gradient Descent")
plt.show()

This graph is very important.

If loss goes down, the model is learning.

If loss goes up, learning rate may be too high.

If loss is flat, learning rate may be too low or model may not be powerful enough.

What Happens If Learning Rate Is Too High?

Suppose learning rate is very large.

Then the model takes huge jumps.

Instead of reaching the minimum, it may jump over it again and again.

Good learning rate:

Loss → ↓ ↓ ↓ ↓ ↓

Too high learning rate:

Loss → ↑ ↓ ↑ ↓ ↑

Code experiment:

lr = 1.0

You may see loss explode.

This is called divergence.

What Happens If Learning Rate Is Too Low?

If learning rate is too small:

lr = 0.000001

The model learns extremely slowly.

Loss decreases, but almost nothing happens for many epochs.

This is why learning rate is one of the most important hyperparameters in deep learning.

Why Gradient Descent Changed Deep Learning

Before deep learning became powerful, one big problem was:

How do we train huge models?

A deep neural network may have:

Millions of weights
Millions of biases
Multiple layers
Complex activations
Huge datasets

Manually choosing weights is impossible.

Trying all combinations is impossible.

Gradient Descent made training possible because it gave a systematic way to improve every parameter.

Even if a model has 10 million parameters, the idea remains:

Find gradient
Move opposite to gradient
Reduce loss
Repeat

That is why Gradient Descent became the engine of deep learning.

What Happens If There Is No Gradient Descent?

Without Gradient Descent, deep learning would almost collapse.

We would have neural networks, but we would not know how to train them efficiently.

Without Gradient Descent:

No automatic weight improvement
No large-scale neural network training
No modern computer vision
No powerful language models
No practical deep learning revolution

We could still use some alternatives like:

Random search
Genetic algorithms
Manual tuning
Closed-form solutions for very small models

But they do not scale like Gradient Descent.

Imagine a neural network with 100 million parameters.

Randomly trying weights would be hopeless.

Gradient Descent gives direction.

That direction changed everything.

Gradient Descent In Neural Networks

In a neural network, every layer has weights.

Input → Hidden Layer → Output

Each layer makes a small transformation.

The final prediction produces loss.

Then backpropagation calculates:

How much each weight contributed to the error

Gradient Descent then updates all weights.

optimizer.zero_grad()
loss.backward()
optimizer.step()

This PyTorch code looks small.

But conceptually:

loss.backward()

calculates gradients.

optimizer.step()

applies Gradient Descent.

That is the heart of deep learning training.

PyTorch Example

import torch
import torch.nn as nn

X = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
y = torch.tensor([[2.0], [4.0], [6.0], [8.0]])

model = nn.Linear(1, 1)

loss_fn = nn.MSELoss()

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for epoch in range(1000):
    y_pred = model(X)

    loss = loss_fn(y_pred, y)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(list(model.parameters()))

This model also learns:

y = 2x

The difference is that PyTorch calculates gradients automatically.

That automatic gradient calculation is called autograd.

Gradient Descent vs Backpropagation

Many beginners confuse these two.

They are related but not the same.

Backpropagation answers:

What are the gradients?

Gradient Descent answers:

How should we update the weights using those gradients?

So:

Backpropagation = gradient calculation

Gradient Descent = parameter update

Together, they train deep neural networks.

Why Deep Learning Needed Gradient Descent More Than Traditional ML

Traditional ML models often have fewer parameters.

Some algorithms do not rely heavily on gradients.

For example:

Decision Trees split data using rules.

KNN stores examples.

Naive Bayes uses probability formulas.

But deep learning is different.

Deep learning is mostly parameter learning.

Millions of parameters must be adjusted.

That is why Gradient Descent became more important in deep learning than almost anywhere else.

Final Thought

Gradient Descent changed deep learning because it converted learning into optimization.

Instead of manually programming intelligence, we define:

Model
Loss Function
Optimizer
Data

Then the model improves itself step by step.

That is the real breakthrough.

Deep learning is not just about big neural networks.

It is about trainable neural networks.

And Gradient Descent is what makes them trainable.

Without it, deep learning would be like a powerful engine with no steering.

With it, random weights slowly become useful knowledge.

Deep Learning Is More Logistic Regression Than You Think

Vineet Chauhan — Sat, 06 Jun 2026 20:01:20 +0000

Why an Algorithm From the 1950s Still Powers Modern AI

When I first learned Machine Learning, I treated Logistic Regression as a beginner algorithm.

You learn it.

You build a classifier.

You get an accuracy score.

Then you move on.

At least that's what I thought.

After Logistic Regression came:

Decision Trees
Random Forests
XGBoost
Neural Networks
Transformers
Large Language Models

The journey seemed straightforward.

Old algorithm → Better algorithm → Even better algorithm.

But after studying Deep Learning more to some extent, I discovered something surprising.

The algorithm I thought I had left behind was everywhere.

Not Decision Trees.

Not Random Forests.

Not SVMs.

Logistic Regression.

And the deeper I looked, the more I realized that modern Deep Learning did not replace Logistic Regression.

It scaled its ideas.

The First Time I Noticed It

I was learning about neural networks.

The instructor drew a neuron:

z = w1*x1 + w2*x2 + b
output = sigmoid(z)

I stared at the equation for a few seconds.

Then it hit me.

That is literally Logistic Regression.

The exact same weighted sum.

The exact same sigmoid activation.

The exact same probability output.

The exact same optimization process.

The only difference?

A neural network has many of them.

What Exactly Does Logistic Regression Do?

At its core, Logistic Regression performs two operations.

Step 1:

Take a weighted sum.

z = w1*x1 + w2*x2 + w3*x3 + b

Step 2:

Convert it into probability.

sigmoid(z) = 1/1+e^-x

The sigmoid function transforms any number into a value between 0 and 1.

Example:

Input = -10 → 0.00004

Input = 0 → 0.5

Input = 10 → 0.99995

This probability becomes the final prediction.

Simple.

Elegant.

Effective.

Now Look At A Neural Network

A neuron performs:

z = w1*x1 + w2*x2 + b

Then:

output = activation(z)

In early neural networks:

activation = sigmoid

which means:

Neuron
=
Logistic Regression Unit

The moment I realized this, neural networks became much easier to understand.

Instead of imagining some magical AI machine, I started seeing thousands of Logistic Regression models stacked together.

Why Not Decision Trees?

This question bothered me for a long time.

Why didn't Deep Learning evolve from Decision Trees?

Why not Random Forests?

Why specifically Logistic Regression?

The answer lies in mathematics.

Reason 1: Logistic Regression Is Differentiable

Decision Trees make hard decisions.

Age > 30 ?

Yes → Left

No → Right

A tiny change in age can suddenly change the entire path.

This creates discontinuities.

Gradient Descent cannot work efficiently.

Logistic Regression is different.

Its sigmoid curve is smooth.

Every tiny change produces a tiny output change.

This makes gradients possible.

And gradients are the fuel of Deep Learning.

Without gradients:

No Backpropagation

Without backpropagation:

No Neural Networks

Without neural networks:

No ChatGPT

Reason 2: Logistic Regression Produces Probabilities

A Decision Tree says:

Class A

Class B

Logistic Regression says:

P(Class A) = 0.92

Probability matters.

Modern AI relies heavily on probabilities.

Examples:

Spam Detection

98% Spam

Medical Diagnosis

73% Cancer Risk

Language Models

P(next word = "cat")

Transformers are fundamentally probability machines.

And Logistic Regression introduced that philosophy long ago.

Reason 3: Cross Entropy Came From Logistic Regression

One of the most important loss functions in Deep Learning is:

L=-[y\log(p)+(1-y)\log(1-p)]

Almost every deep learning engineer uses it.

Image Classification.

Medical AI.

Fraud Detection.

NLP.

Large Language Models.

The interesting part?

This is the same loss function used in Logistic Regression.

The entire deep learning world still depends on it.

Reason 4: Logistic Regression Is A Single Neuron

This was the biggest realization for me.

A Logistic Regression model can be represented as:

Input
 ↓
Weighted Sum
 ↓
Sigmoid
 ↓
Output

Now look at a neural network:

Input
 ↓
100 Neurons
 ↓
100 Neurons
 ↓
100 Neurons
 ↓
Output

Each neuron is doing a very similar operation.

The network becomes powerful because thousands of these simple units collaborate.

Deep Learning is not complexity replacing simplicity.

It is simplicity repeated at scale.

Let's Verify This With Code

Logistic Regression

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()

model.fit(X_train, y_train)

predictions = model.predict(X_test)

Now a neural network.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10,1),
    nn.Sigmoid()
)

Look carefully.

Both perform:

Weighted Sum
Sigmoid Transformation
Probability Prediction

Mathematically they are nearly identical.

The PyTorch version is essentially Logistic Regression implemented as a neural network.

The Real Difference

If they are so similar, why use Deep Learning?

Because Logistic Regression can only learn simple boundaries.

Imagine separating red and blue dots.

Logistic Regression creates:

One Straight Line

Deep Learning creates:

Curves
Shapes
Complex Regions
Non-Linear Patterns

By stacking layers, the network gradually transforms simple linear boundaries into highly complex decision surfaces.

That is the true power of Deep Learning.

Not a different idea.

A larger version of the same idea.

The Most Surprising Place I Found Logistic Regression

LSTMs.

The architecture behind many sequence models.

Inside every LSTM cell are gates.

Forget Gate.

Input Gate.

Output Gate.

Guess what activation function they use?

Sigmoid.

Every gate computes probabilities.

Every gate decides:

Keep Information?

Forget Information?

using Logistic Regression principles.

Even modern AI systems still carry its DNA.

Final Thoughts

When I first learned Logistic Regression, I thought it was something to finish and forget.

Now I see it differently.

I see it as the first neural network.

I see it as the origin of probability-based learning.

I see it as the mathematical foundation behind cross entropy, gradient descent, and backpropagation.

The next time someone says Logistic Regression is an old algorithm, remember:

Deep Learning did not replace Logistic Regression.

Deep Learning scaled it.

And some of the most advanced AI systems ever built still rely on ideas introduced by Logistic Regression decades ago.

Better Data Beats Better Algorithms: Before Changing the Model, Change the Data

Vineet Chauhan — Sat, 06 Jun 2026 19:47:54 +0000

How Feature Engineering Taught Me That Better Data Often Beats Better Algorithms

When I first started learning Machine Learning, I believed what many beginners believe:

If my model is not performing well, I need a better algorithm.

So I kept switching models.

I moved from Logistic Regression to Decision Trees, then Random Forest, and later even started reading about XGBoost and Neural Networks.

The results improved slightly, but never dramatically.

What surprised me was that the biggest improvement didn't come from changing the algorithm.

It came from changing the data.

The Problem

I was working on a dataset containing missing values, outliers, and categorical variables.

Like many beginners, my first instinct was simple:

model.fit(X_train, y_train)
pred = model.predict(X_test)

The model trained successfully.

The accuracy looked acceptable.

But something felt wrong.

The data itself was messy.

Some columns contained missing values.

Some numerical features had extreme outliers.

Several categorical columns were represented as text.

Yet I expected the model to magically learn everything.

My First Experiment

I trained a Logistic Regression model on the raw dataset.

Results:

Accuracy : 72%

Not terrible.

Not impressive either.

Instead of changing the model, I decided to investigate the data.

This turned out to be the most important decision of the entire project.

Step 1: Handling Missing Values

The dataset contained several missing values.

At first I considered simply deleting rows.

df.dropna(inplace=True)

The problem?

I lost a significant portion of the data.

So I experimented with multiple approaches:

Mean Imputation

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy='mean')
X = imputer.fit_transform(X)

Median Imputation

imputer = SimpleImputer(strategy='median')

KNN Imputation

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)
X = imputer.fit_transform(X)

KNN preserved relationships between records much better than simple averaging.

This alone improved performance.

Step 2: Fighting Outliers

I then visualized the numerical columns.

The boxplots looked terrible.

A few extreme values were stretching entire distributions.

sns.boxplot(df["experience"])

The model was spending too much effort trying to fit a handful of unusual observations.

I used IQR-based treatment.

Q1 = df["experience"].quantile(0.25)
Q3 = df["experience"].quantile(0.75)

IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

df = df[(df["experience"] >= lower) &
        (df["experience"] <= upper)]

After removing outliers, the data distribution became much cleaner.

More importantly, the model began learning actual patterns instead of noise.

Step 3: Encoding Categorical Features

Machine Learning algorithms cannot understand text.

They only understand numbers.

So columns like:

Male
Female

Private
Public

Graduate
Masters

needed transformation.

I applied One-Hot Encoding.

pd.get_dummies(df,
               columns=["gender",
                        "company_type"])

and Ordinal Encoding where order mattered.

education_level

High School
Graduate
Masters
PhD

This converted human-readable categories into machine-readable information.

Step 4: Feature Scaling

Some columns ranged between:

0 – 5

while others ranged between:

0 – 100000

Distance-based algorithms become biased toward larger values.

I applied MinMax Scaling.

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Now every feature contributed fairly.

What Happened Next?

I trained the exact same Logistic Regression model again.

Nothing changed except the data.

Results:

Before Feature Engineering : 72%

After Feature Engineering  : 86%

A gain of 14 percentage points.

Without changing the algorithm.

Without using deep learning.

Without adding complexity.

Just by improving the data.

The Most Important Lesson

This project changed the way I think about Machine Learning.

Earlier I believed:

Better Algorithm
       ↓
Better Results

Now I believe:

Better Data
       ↓
Better Features
       ↓
Better Results

Most real-world machine learning problems are not algorithm problems.

They are data problems.

A powerful model trained on poor-quality data will still struggle.

A simple model trained on clean, meaningful data can often outperform much more complex alternatives.

Challenges I Faced

The hardest part was not training the model.

The hardest part was preparing the data.

Some difficulties included:

Losing rows during Complete Case Analysis
Choosing between Mean, Median, and KNN Imputation
Combining transformed datasets
Handling dimensionality after One-Hot Encoding
Identifying genuine outliers versus valuable rare cases

These challenges taught me more than model training ever did.

Final Thoughts

Feature Engineering is not the most glamorous part of Machine Learning.

Nobody posts screenshots of missing value treatment on social media.

Nobody celebrates scaling features.

Yet this is where much of the real improvement happens.

After this project, I stopped asking:

Which model should I use?

and started asking:

What is my data trying to tell me?

That single change in mindset improved my machine learning skills more than learning any new algorithm.

Feature Engineering is Not Just “Cleaning Data”: What I Learned While Building a Real ML Pipeline

Vineet Chauhan — Thu, 28 May 2026 10:43:45 +0000

Most machine learning tutorials make preprocessing look straightforward.

Handle missing values.
Encode categorical features.
Train the model.
Get accuracy.

But while working on a real classification dataset, I realized feature engineering is far less about applying textbook techniques and far more about making careful decisions under uncertainty.

This project completely changed how I think about preprocessing.

Instead of writing another “complete guide to feature engineering,” I wanted to document the actual engineering problems I faced while building a preprocessing pipeline — including debugging mistakes, failed assumptions, distribution shifts, encoding challenges, and how preprocessing itself changed model behaviour.

The Dataset Looked Simple at First

Initially, the dataset looked manageable:

Numerical features
Categorical features
Missing values
Binary target variable

Nothing seemed unusual.

But the moment preprocessing started, the real complexity appeared.

The First Problem: Missing Values Were Uneven Everywhere

One of the first things I checked was the percentage of missing values across columns.

Some columns had:

less than 1% missing values
some had 5–10%
others had more than 30%

This immediately raised an important question:

Should every missing value be handled using the same strategy?

The answer quickly became no.

My Initial Mistake: Applying One Strategy Everywhere

At first, I tried treating all missing values similarly.

That approach failed quickly because:

Complete Case Analysis removed too many rows
KNN Imputation behaved poorly on categorical-heavy features
Encoded categorical values introduced unrealistic numeric relationships
Feature distributions started changing unexpectedly

This was the first moment I realized:

Feature engineering is not a fixed recipe.

Different features require different preprocessing decisions.

Using Complete Case Analysis (CCA)

For columns with less than 5% missing values, I used Complete Case Analysis.

At first, this seemed harmless.

But then I decided to compare feature distributions before and after row deletion.

This turned out to be one of the most important observations in the project.

Even small row deletions slightly changed feature density and distributions.

That was the moment I understood:

Missing value handling is not only about removing NaNs — it can also reshape the dataset itself.

A Small Pandas Mistake That Broke My Pipeline

One debugging issue confused me for quite a while.

Initially, I wrote:

df[cols].dropna()

This unintentionally removed all other columns from the dataframe.

The correct approach was:

df.dropna(subset=cols)

The difference was tiny syntactically but huge logically.

This taught me something surprisingly important:

Many machine learning problems are not model problems.
They are dataframe manipulation problems.

Why KNN Imputer Became Complicated

Initially, I planned to use KNN Imputer for all remaining missing values.

But another issue appeared immediately.

KNN relies on distance calculations.

Distance works naturally for numerical data, but categorical columns require encoding first.

That introduced several complications:

Label encoding created artificial numeric relationships
One Hot Encoding exploded feature dimensionality
NaN values converted into strings accidentally during preprocessing
Encoded categories distorted neighbor similarity

This made me realize:

KNN Imputer works much better for numerical features than heavily categorical datasets.

Eventually, I switched to a hybrid preprocessing strategy instead of forcing one solution everywhere.

Final Missing Value Strategy

I ended up using:

Feature Type:- Strategy:-

Low missing numerical ---> Complete Case Analysis
High missing numerical ---> Median/KNN Imputation
High missing categorical ---> Most Frequent / “Missing” category

This hybrid approach worked far better than blindly applying one technique globally.

Encoding Was More Important Than I Expected

Encoding looked simple in theory.

But in practice, deciding between:

One Hot Encoding
Ordinal Encoding
Label Encoding

actually mattered a lot.

I used:

One Hot Encoding

For:

gender
major_discipline
company_type
enrolled_university

because these categories had no natural order.

Ordinal Encoding

For:

education_level

because educational levels actually contain ranking:

Primary School < High School < Graduate < Masters < PhD

This distinction improved model behavior more than I initially expected.

The Most Interesting Observation: Preprocessing Changed the Models More Than the Models Changed Themselves

I trained multiple models:

Logistic Regression
Decision Tree
Random Forest

both before and after preprocessing.

The results were surprisingly different.

Linear models improved heavily after scaling and proper encoding.

Random Forest remained comparatively stable even before aggressive preprocessing.

That observation completely changed my perspective.

Data preprocessing often influences performance more than changing the algorithm itself.

Another Real Problem: Feature Explosion

After applying One Hot Encoding, the number of features increased rapidly.

This was another practical challenge rarely discussed in beginner tutorials.

Encoding solved categorical representation issues, but it also increased dimensionality and preprocessing complexity significantly.

Why Pipelines Became Necessary

At one point, preprocessing became chaotic.

Different transformations were happening separately:

scaling
imputation
encoding
train-test transformations

Tracking transformed columns manually became painful.

This was when I finally understood why sklearn Pipelines and ColumnTransformers matter so much.

Not because they look advanced —
but because preprocessing becomes unmanageable very quickly in real projects.

What This Project Changed for Me

Before this project, I thought feature engineering mostly meant:

removing null values
encoding categories
scaling features

Now I think differently.

Feature engineering is closer to:

understanding how data behaves under transformation.

Every preprocessing decision changes:

distributions
feature relationships
dimensionality
information retention
model assumptions

Even small preprocessing choices can significantly change model behavior.

*Final Thoughts
*
One thing became very clear after this project:

Machine learning is not just model training.

Most real effort goes into:

understanding data
debugging preprocessing
handling edge cases
preserving useful information
testing assumptions

Feature engineering is where datasets stop behaving like clean classroom examples and start behaving like real systems.

"And honestly, that is where machine learning starts becoming interesting."

Building a Hantavirus Misinformation Detector: Challenges of NLP in Low-Data Health Domains

Vineet Chauhan — Sat, 16 May 2026 17:00:47 +0000

Most fake news detection projects rely on massive datasets containing thousands of examples.

I wanted to explore something much more difficult:

Can a small NLP system detect misinformation around an emerging disease like Hantavirus?

What made this project interesting was not the model itself, but the challenge of working in a low-data environment where reliable misinformation examples barely exist.

Unlike COVID-19 misinformation datasets, hantavirus-related misinformation is extremely limited online. This forced me to manually curate both factual and misleading claims while understanding how health misinformation behaves linguistically.

This project became less about achieving high accuracy and more about:

understanding NLP pipelines,
handling imperfect datasets,
and analyzing misinformation patterns.

2. Understanding the Problem

Health misinformation spreads differently from normal fake news.

Many misleading claims are:

1. partially believable,
2. emotionally framed,
3. or based on incomplete truths.

Examples:

“Natural remedies can cure hantavirus”
“Governments are hiding outbreak data”
“Hot water prevents infection”

The challenge was not simply classifying text as fake or real, but understanding how subtle misinformation patterns emerge in health-related discussions.

3. Dataset Creation (The Hardest Part)

This was by far the most difficult stage of the project.

Unlike mainstream misinformation domains, there are very few structured datasets specifically related to hantavirus misinformation. Because of this, I manually curated a small dataset using:

trusted medical sources,
news articles,
and realistic misinformation patterns.

4. Real Data Sources

I collected factual information from:

WHO
CDC
Reuters

Examples included:

transmission details,
symptoms,
prevention methods,
and treatment limitations.

5. Fake Data Construction

Finding real misinformation examples for hantavirus was difficult because the topic is relatively niche.

Instead of generating random false statements, I focused on realistic misinformation patterns commonly seen in health-related fake news:

miracle cures,
conspiracy theories,
exaggerated transmission claims,
and misleading prevention methods.

Examples:

“Garlic water can completely cure hantavirus”
“The virus spreads rapidly through city air systems”
“A secret vaccine already exists”

Dataset Structure

The dataset included:

text
label
source
category
difficulty

This structure helped organize misinformation types and analyze which claims were easier or harder for the model to classify.

7. Dataset Analysis

Fake vs Real Distribution

Category Distribution

Difficulty Distribution

8. NLP Pipeline

The NLP pipeline was intentionally kept simple to better understand the fundamentals.

The workflow consisted of:

Text preprocessing
TF-IDF vectorization
Logistic Regression classification

9. Text Preprocessing

The first step involved cleaning the text data:

converting text to lowercase,
removing punctuation,
removing unnecessary spaces,
and standardizing sentence structure.

10. TF-IDF Vectorization

Machine learning models cannot directly understand raw text.

TF-IDF converts words into numerical representations based on their importance across the dataset.

This allowed the model to identify patterns such as:

“secret cure”
“government hiding”
“supportive care”
“WHO reports”

11. Most Interesting Observation

One of the most surprising findings was:

believable misinformation is much harder to classify than extreme misinformation.

Claims like:

“Herbal remedies may reduce hantavirus symptoms”

were more difficult for the model than clearly absurd claims.

This highlighted an important limitation of simple NLP models:
they rely heavily on statistical language patterns rather than true medical understanding.

12. Limitations

This project has several limitations:

small dataset size,
manually curated misinformation,
limited real-world social media data,
and no deep learning models.

Because of these constraints, the model should not be treated as a production-ready misinformation detector.

Instead, this project should be viewed as:

an exploratory NLP experiment in a low-data health misinformation domain.

13. Future Improvements

There are several directions for improving this project:

collecting real social media misinformation,
increasing dataset size,
using transformer-based models like BERT,
multilingual misinformation detection,
and explainable AI methods such as SHAP or LIME.

14. Final Thoughts

This project taught me that the hardest part of NLP is often not the model itself.

It is:

collecting meaningful data,
understanding ambiguity,
and dealing with imperfect real-world information.

Working on a low-data problem like hantavirus misinformation made the project far more challenging — and far more educational — than simply training a model on a large public dataset.

Even though the model itself was simple, the process revealed how difficult health misinformation detection actually is in practice