DEV Community: Vineet Chauhan

The Invisible Complexity Behind Simple Human Actions: Why Teaching Machines What We Do Naturally Is So Hard

Vineet Chauhan — Tue, 04 Aug 2026 18:41:52 +0000

The Biggest Thing I Realized After Spending One Year Learning AI

For the last one year, I've spent most of my time studying Machine Learning, Deep Learning, Generative AI, Agentic AI, and Data Science. Along with that came hours of struggling with Probability, Statistics, Linear Algebra, Calculus, and all the mathematics behind these models.

When I started, my questions were pretty simple: How does Linear Regression actually work? Why does Gradient Descent find the minimum? What's the intuition behind Neural Networks? I was focused on understanding individual algorithms.

But over the last couple of months, I realized something much bigger.

We're not just building software—we're trying to recreate one of the most extraordinary abilities in nature: intelligence.

Think about something as simple as recognizing a friend. We look at a face for a fraction of a second and instantly know who it is, what emotion they're showing, and even the situation they're might be in. Our brain does this so naturally that we never think about how difficult it actually is.

Now ask a machine to do the same.

For us, it's effortless. For a machine, it's millions or even billions of mathematical operations, huge datasets, complex neural networks, optimization algorithms, and massive computational power—all just to perform a task that a child can do without thinking.

That's the moment AI truly blew my mind.

Every time I learn something new, I appreciate even more the incredible engineering and research happening behind companies like OpenAI and Anthropic. What looks like a simple chatbot response or image generation is actually the result of years of research, mathematics, and thousands of brilliant minds working together.

The more I learn about AI, the more I realize we're not just teaching machines to solve problems—we're trying to teach them to understand the world the way humans do. And honestly, that's one of the greatest engineering challenges humanity has ever taken on.

From Learning Machine Learning to Competing on Kaggle: My First End-to-End Playground Competition Journey

Vineet Chauhan — Thu, 30 Jul 2026 09:16:23 +0000

How I applied Exploratory Data Analysis, Feature Engineering, Pipelines, and Ensemble Models to solve a real-world machine learning problem—and the lessons I learned along the way.

Introduction

There comes a point in every machine learning learner's journey when watching tutorials and completing small practice exercises are no longer enough.

After spending weeks understanding statistics, exploratory data analysis (EDA), feature engineering, preprocessing techniques, and classical machine learning algorithms, I wanted to answer one question:

Can I apply everything I've learned to a real machine learning competition?

That's when I decided to participate in a Kaggle Playground competition.

Unlike classroom datasets, Kaggle competitions force you to think like a machine learning engineer. You're responsible for understanding messy data, building preprocessing pipelines, selecting models, evaluating performance, debugging errors, and finally creating a submission that competes with thousands of participants.

This article documents my complete journey—from loading the dataset to building production-style preprocessing pipelines and training multiple ensemble models. Along the way, I'll also share the challenges I faced, what worked well, and the lessons I'll carry into future competitions.

Why Kaggle?

Learning machine learning isn't just about knowing algorithms.

Real-world ML requires answering questions like:

Which features are useful?
How should missing values be handled?
Should categorical variables be one-hot encoded or ordinal encoded?
Which preprocessing steps belong inside a pipeline?
How do different ensemble models compare?

Kaggle provides an environment where all of these questions matter.

Instead of building a model that works only inside a notebook, you're solving a problem under realistic constraints and evaluating your solution on unseen data.

Competition Goal

The objective of this Playground competition was to predict the target class based on a combination of numerical and categorical features related to health and lifestyle.

The workflow followed the same structure used in many real-world machine learning projects.

Raw Dataset
      │
      ▼
Exploratory Data Analysis
      │
      ▼
Missing Value Analysis
      │
      ▼
Feature Engineering
      │
      ▼
Preprocessing Pipeline
      │
      ▼
Model Training
      │
      ▼
Evaluation
      │
      ▼
Prediction
      │
      ▼
Kaggle Submission

Although the workflow appears simple, every stage requires careful decision-making.

Understanding the Dataset

The first step was loading both the training and testing datasets using Pandas.

train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

Before writing a single machine learning model, I spent time understanding the structure of the data.

Some of the questions I explored were:

How many rows and columns are present?
Which features are numerical?
Which features are categorical?
Are there missing values?
What is the distribution of each feature?
Are there any obvious outliers?

Skipping this stage often leads to poor model performance because preprocessing decisions depend entirely on the characteristics of the data.

Exploratory Data Analysis (EDA)

EDA turned out to be one of the most valuable parts of the project.

Instead of immediately training a model, I wanted to understand how the dataset behaved.

I separated the features into two broad categories:

Numerical Features

Examples included:

Sleep Duration
Heart Rate
BMI
Calorie Expenditure
Step Count
Exercise Duration
Water Intake

Categorical Features

Examples included:

Diet Type
Gender
Smoking/Alcohol
Physical Activity Level
Sleep Quality
Stress Level

This separation made it much easier to apply different visualization techniques.

Visualizing Numerical Features

For numerical columns, I explored the data using:

Histograms
KDE Plots
Boxplots

These visualizations helped answer important questions.

Are the features normally distributed?

Some variables showed distributions close to normal, while others exhibited noticeable skewness.

Are there extreme values?

Boxplots revealed the presence of outliers in several numerical features.

Is the data symmetric?

Using skewness calculations together with KDE plots made it easier to understand whether transformations might be useful.

Instead of blindly preprocessing the data, these visualizations allowed me to make decisions based on evidence.

Understanding Missing Values

One of the first analyses I performed was checking the percentage of missing values across the dataset.

train.isnull().mean() * 100

Missing values are one of the most common problems in real-world datasets.

Ignoring them isn't an option because most machine learning algorithms cannot train with incomplete data.

Rather than replacing every missing value with a single constant, I decided to use appropriate preprocessing techniques later in the pipeline.

Exploring Categorical Features

For categorical variables, I created count plots to understand the frequency distribution of each category.

These plots answered questions like:

Which diet type appears most frequently?
How are physical activity levels distributed?
Does gender influence certain lifestyle variables?
Are some categories extremely rare?

I also explored relationships between different categorical variables using grouped visualizations.

This helped me develop an intuition about the dataset before training any model.

Building Intuition Before Building Models

One lesson became very clear during EDA:

Better understanding often leads to better preprocessing.

Machine learning isn't just about choosing a powerful algorithm.

The quality of the input data has an enormous impact on the quality of the final predictions.

That's why I spent significant time exploring the data before moving on to feature engineering.

Feature Engineering

After understanding the dataset, the next step was preparing it for machine learning.

The first task was separating features and the target variable.

X = train.drop(columns=["health_condition"])
y = train["health_condition"]

This created a clear distinction between:

Input features (X)
Target labels (y)

I then split the training data into training and validation sets.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

Creating a validation set allowed me to evaluate model performance before making a Kaggle submission.

Why Preprocessing Matters

Different feature types require different preprocessing strategies.

For example:

Feature Type	Preprocessing
Numerical	Imputation + Scaling
Ordinal Categorical	Ordinal Encoding
Nominal Categorical	One-Hot Encoding
Target Variable	Label Encoding

Applying the wrong preprocessing technique can significantly reduce model performance.

Instead of manually transforming every column, I wanted a cleaner and more reusable solution.

That's where Scikit-learn Pipelines became extremely useful.

Organizing Features

I divided the features into logical groups.

Numerical Columns

Continuous numerical variables requiring scaling and missing value handling.

Ordinal Categories

Features with a natural ordering, such as activity or stress levels.

These can be safely converted into ordered numerical values.

Nominal Categories

Features such as gender or diet type have no natural ordering.

Assigning arbitrary numbers could introduce unintended relationships, so One-Hot Encoding is a better choice.

This separation made the preprocessing pipeline much easier to maintain.

Building a Production-Style Preprocessing Pipeline

Rather than writing preprocessing code repeatedly for every model, I used Scikit-learn's Pipeline and ColumnTransformer.

This approach provides several advantages:

Cleaner code
Reduced risk of data leakage
Consistent preprocessing
Easier experimentation with multiple models
Production-ready workflow

Instead of manually applying transformations one by one, the pipeline automatically performs every preprocessing step before model training.

This is one of the biggest improvements I made compared to my earlier machine learning projects.

Building a Production-Ready Machine Learning Pipeline

One of the biggest improvements I made in this project compared to my earlier machine learning exercises was moving away from manual preprocessing.

Instead of writing separate preprocessing code for every model, I built a reusable pipeline using Scikit-learn.

This approach follows the same philosophy used in production machine learning systems.

Rather than remembering dozens of preprocessing steps, everything is automated inside a single workflow.

Designing the Preprocessing Pipeline

Different feature types require different transformations.

My preprocessing workflow looked like this:

Raw Dataset
      │
      ▼
Separate Numerical & Categorical Features
      │
      ├───────────────┐
      │               │
      ▼               ▼
 Numerical        Categorical
      │               │
KNN Imputer    Simple Imputer
      │               │
StandardScaler  Encoding
      │               │
      └──────┬────────┘
             ▼
     ColumnTransformer
             ▼
     Machine Learning Model

Instead of transforming every column manually, the pipeline automatically applied the correct preprocessing to each feature type.

This made the code cleaner, reusable, and less error-prone.

Handling Missing Values

Missing values are unavoidable in real-world datasets.

Instead of dropping rows and losing valuable information, I applied different imputation strategies depending on the feature type.

For numerical features, I used KNN Imputer.

The idea behind KNN Imputation is simple.

Instead of filling missing values with a mean or median, it looks for the most similar observations and estimates the missing value based on those neighbors.

For categorical features, I used Simple Imputer with the most frequent category.

This preserved the integrity of categorical data while avoiding unnecessary data loss.

Encoding Categorical Variables

Machine learning models cannot understand text directly.

Therefore, categorical variables needed to be converted into numerical representations.

I divided categorical columns into two groups.

Ordinal Features

These have a natural order.

Examples include activity levels or stress levels.

For these variables, I used Ordinal Encoding.

Low      → 0
Medium   → 1
High     → 2

Since these categories have meaningful ordering, ordinal encoding preserves that relationship.

Nominal Features

Some features have no natural ordering.

Examples include gender or diet type.

Assigning numbers such as

Male = 0
Female = 1

could unintentionally imply a mathematical relationship.

Instead, I applied One-Hot Encoding, where each category receives its own binary column.

This prevents the model from assuming nonexistent order.

Why I Used ColumnTransformer

Without a ColumnTransformer, preprocessing quickly becomes difficult to manage.

Different feature groups require different transformations.

Instead of manually applying each transformation, ColumnTransformer allows everything to happen in one unified preprocessing stage.

This makes experimentation much easier because changing the model no longer requires rewriting preprocessing code.

Building Reusable Pipelines

After creating the preprocessing pipeline, I combined it with different machine learning algorithms.

Instead of writing separate preprocessing code for every model, I simply replaced the final estimator.

Conceptually, every pipeline followed this structure:

Preprocessing
        │
        ▼
 Machine Learning Model
        │
        ▼
 Predictions

This allowed me to compare multiple algorithms while keeping preprocessing completely consistent.

Training Multiple Ensemble Models

Rather than relying on a single algorithm, I trained several ensemble models to compare their performance.

The models included:

Random Forest
XGBoost
CatBoost
LightGBM

Each model has its own strengths.

Random Forest

Random Forest was my baseline ensemble model.

Its advantages include:

Robust performance
Handles nonlinear relationships
Resistant to overfitting
Useful for feature importance

It gave me a strong benchmark before experimenting with boosting algorithms.

XGBoost

XGBoost is one of the most popular gradient boosting libraries in competitive machine learning.

In my implementation, I configured parameters such as:

300 estimators
Learning rate of 0.05
Maximum tree depth
Subsampling
Column sampling
Histogram-based tree construction
GPU acceleration

The goal was to balance predictive performance with computational efficiency.

CatBoost

CatBoost is particularly effective when working with datasets containing categorical variables.

Even though my preprocessing pipeline already handled encoding, CatBoost still provided another strong ensemble model for comparison.

I trained it using GPU acceleration to reduce computation time.

LightGBM

LightGBM is designed for efficiency.

It uses histogram-based learning and leaf-wise tree growth, making it extremely fast while maintaining competitive performance.

Its ability to scale efficiently makes it a popular choice in Kaggle competitions.

Evaluating Model Performance

Training a model is only half of the workflow.

The next step is evaluating how well it performs.

I compared the models using several evaluation metrics instead of relying on accuracy alone.

These included:

Accuracy
Precision
Recall
F1 Score

Each metric provides different insights into model behavior.

Looking at multiple metrics helps avoid misleading conclusions that can arise from using only accuracy.

Classification Reports

To gain a deeper understanding of performance, I generated classification reports for each model.

These reports show:

Precision for every class
Recall for every class
F1-score
Overall accuracy
Macro average
Weighted average

Instead of treating the model as a black box, these reports reveal where it performs well and where it struggles.

Confusion Matrix

Another valuable visualization was the confusion matrix.

Rather than simply knowing that a prediction was incorrect, the confusion matrix shows which classes the model confuses with one another.

This often reveals patterns that raw accuracy cannot.

For multiclass classification problems, confusion matrices provide valuable insights into model weaknesses.

Comparing Models

Instead of evaluating models separately, I created a comparison table containing metrics such as:

Model	Accuracy	Precision	Recall	F1 Score
XGBoost	Compared	Compared	Compared	Compared
CatBoost	Compared	Compared	Compared	Compared
LightGBM	Compared	Compared	Compared	Compared

A comparison table makes it much easier to identify trade-offs between different algorithms.

Lessons I Learned

Looking back, this competition taught me far more than simply training machine learning models.

Some of the most important lessons were:

Understanding the data matters more than choosing the model.

A strong preprocessing pipeline often improves performance more than switching algorithms.

Pipelines save enormous amounts of time.

Instead of rewriting preprocessing code repeatedly, everything becomes modular and reusable.

Evaluation should never rely on a single metric.

Accuracy alone rarely tells the complete story.

Ensemble methods are powerful.

Comparing multiple algorithms helped me understand the strengths and limitations of each approach.

Challenges Along the Way

Like every real project, this competition involved several debugging sessions.

I encountered issues related to:

preprocessing
pipeline construction
model experimentation
evaluation
submission workflow

Although these challenges were sometimes frustrating, solving them significantly improved my understanding of the complete machine learning lifecycle.

In hindsight, those debugging sessions taught me as much as the successful model training itself.

Beyond a Kaggle Score

One of the biggest misconceptions about Kaggle is that success is measured only by leaderboard position.

For me, the real achievement wasn't just generating predictions.

It was learning how to build an end-to-end machine learning workflow.

This project required me to combine concepts that I had previously learned separately:

Exploratory Data Analysis
Data Cleaning
Feature Engineering
Missing Value Handling
Encoding
Scaling
Pipelines
Ensemble Learning
Model Evaluation

For the first time, these concepts came together in a single project.

Final Thoughts

This competition marked an important milestone in my machine learning journey.

Before this project, preprocessing techniques, pipelines, ensemble models, and evaluation metrics were topics I had studied individually.

Working through a real Kaggle competition showed me how these pieces fit together to solve an end-to-end machine learning problem.

While there is still plenty to learn—from advanced feature engineering and hyperparameter optimization to model ensembling and deployment—this experience gave me confidence that I can move beyond tutorials and apply machine learning concepts in practical settings.

If you're just beginning your Kaggle journey, my biggest advice is simple:

Don't chase the leaderboard first. Chase understanding.

A solid foundation in data analysis, preprocessing, and experimentation will take you much further than memorizing model parameters. Every competition is an opportunity to learn, refine your workflow, and become a better machine learning practitioner.

For me, this wasn't just another notebook—it was the project that transformed weeks of studying into real, hands-on experience. And that's exactly what makes Kaggle such a valuable platform for anyone serious about learning machine learning.

What Really Happens When You Search on Netflix, Spotify, or Amazon? Behind the Scenes of Modern Search Systems

Vineet Chauhan — Fri, 03 Jul 2026 14:49:12 +0000

We type a few words. Millions of computations happen.

Every day, billions of people search for something.

"Interstellar" on Netflix
"Shape of You" on Spotify
"Wireless Mouse" on Amazon

The results appear almost instantly.

To most users, it feels simple. But behind that tiny search box is one of the most sophisticated engineering systems ever built.

Search is no longer just about finding matching words. Modern platforms try to understand what you mean, predict what you're looking for, personalize the results, rank millions of possibilities, and deliver everything within a fraction of a second.

In this article, we'll follow the complete journey of a search query—from the moment you press Enter to the instant the perfect result appears on your screen.

Step 1: You Type a Query

Imagine you open Amazon and type:

wireless gaming mouse

Your computer sends this request to Amazon's servers.

User
   │
   ▼
Search Box
   │
   ▼
Amazon Search Server

At this point, the system doesn't simply compare text with product names.

Instead, it starts understanding the query itself.

Step 2: Understanding the Query

Humans naturally understand language.

Computers don't.

So the first task is query preprocessing.

The search engine breaks your sentence into meaningful pieces.

Original Query

Wireless Gaming Mouse

After preprocessing:

wireless
gaming
mouse

Then several operations happen.

Convert to lowercase

Wireless → wireless

Remove unnecessary words

"The Best Wireless Mouse"

↓

best
wireless
mouse

Correct spelling

wirless

↓

wireless

Understand synonyms

TV

↓

Television

Shoes

↓

Sneakers

Amazon, Netflix and Spotify all maintain massive synonym dictionaries.

Step 3: Searching the Index

Suppose Amazon has 500 million products.

Would it compare your query against every product?

Absolutely not.

That would take far too long.

Instead, search engines build something called an Inverted Index.

Think of it like the index at the back of a textbook.

Instead of reading every page, you jump directly to the pages containing a keyword.

Example:

wireless

↓

Product 4
Product 18
Product 52
Product 190

gaming

↓

Product 18
Product 77
Product 190

mouse

↓

Product 18
Product 190
Product 250

The engine quickly finds products containing all three terms.

This reduces the search space from millions of products to only a few thousand.

Step 4: Keyword Matching Isn't Enough

Imagine searching for

Laptop Stand

A product named

Adjustable Notebook Holder

might never appear.

Even though both mean almost the same thing.

Traditional keyword search struggles here because the words are different.

Modern search engines solve this using semantic search.

Step 5: Understanding Meaning with Embeddings

AI converts every word, sentence or product into a list of numbers called an embedding.

Instead of storing words...

Dog

the AI stores something like

[0.42, -0.81, 1.55, ...]

Words with similar meanings appear close together inside this mathematical space.

Dog -------- Puppy

Cat -------- Kitten

Movie -------- Film

Laptop -------- Notebook

Now when someone searches

Notebook

the system can still recommend laptops because their embeddings are close together.

This is why modern search feels much smarter than searching a PDF with Ctrl + F.

Practical Example in Python

Using Sentence Transformers, we can perform semantic search in just a few lines.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer("all-MiniLM-L6-v2")

products = [
    "Wireless Gaming Mouse",
    "Mechanical Keyboard",
    "Gaming Laptop",
    "Bluetooth Speaker"
]

query = "Wireless Mouse"

product_embeddings = model.encode(products)
query_embedding = model.encode([query])

scores = cosine_similarity(query_embedding, product_embeddings)

for product, score in zip(products, scores[0]):
    print(product, round(score,3))

Instead of matching exact words, the model compares meaning.

Step 6: Ranking the Results

Suppose the search finds 5,000 matching products.

Should they all appear randomly?

Of course not.

The system now ranks them.

Some ranking signals include:

Relevance
Popularity
Ratings
Reviews
Purchase history
Availability
Delivery speed
User preferences
Click-through rate
Conversion rate

A simplified ranking formula looks like:

Final Score

=

0.4 × Relevance

+

0.3 × Popularity

+

0.2 × Rating

+

0.1 × Personalization

The highest-scoring products appear first.

Step 7: Personalization

Now comes one of the most powerful components.

The same search query can produce completely different results for different users.

Suppose two people search:

Shoes

User A

Previously searched:

Football
Nike
Adidas

Amazon is likely to recommend football boots.

User B

Previously searched:

Running
Marathon
Sports Watch

Amazon is more likely to recommend running shoes.

The query is identical.

The user is different.

So the results are different.

Step 8: Recommendation Systems

Search isn't the only way users discover content.

Recommendations are equally important.

Netflix recommends movies.

Spotify recommends songs.

Amazon recommends products.

These systems use several techniques:

Collaborative Filtering

People with similar interests often enjoy similar items.

If thousands of users who watched

Interstellar

also watched

Inception

Netflix learns this relationship.

Content-Based Filtering

Recommend similar items.

Action movie

↓

More action movies.

Rock song

↓

More rock songs.

Gaming laptop

↓

Gaming accessories.

Hybrid Recommendation

Most companies combine multiple recommendation methods with machine learning models.

This provides significantly better recommendations than relying on a single technique.

Step 9: Why Results Are Instant

Imagine rebuilding the ranking every time someone searches.

It would be far too slow.

Companies use multiple optimization techniques:

Caching
Distributed databases
Load balancing
Parallel processing
Search indexes
Vector databases
Content Delivery Networks (CDNs)

These systems reduce search latency from seconds to milliseconds.

Real Architecture

User

↓

Search API

↓

Query Processing

↓

Keyword Search

+

Semantic Search

↓

Ranking Engine

↓

Recommendation Engine

↓

Personalization Layer

↓

Final Results

Each block may itself contain dozens of microservices working together.

Why Vector Databases Matter

Traditional databases answer questions like:

Find all products costing less than ₹2000.

Vector databases answer questions like:

Find products that are semantically similar to this product.

Popular vector databases include:

FAISS
Pinecone
Milvus
Weaviate
ChromaDB
Qdrant

They power modern AI search and Retrieval-Augmented Generation (RAG) systems.

Where Machine Learning Fits In

Machine learning improves search in many ways:

Spell correction
Query understanding
Ranking
Recommendations
Fraud detection
Personalized search
Demand prediction

The goal isn't just to find a matching result—it's to predict the result you're most likely to choose.

Key Takeaways

Modern search systems are much more than databases with a search bar. They combine information retrieval, natural language processing, recommendation systems, machine learning, distributed computing, and efficient data structures to deliver highly relevant results in milliseconds.

The next time you search for a movie on Netflix, a song on Spotify, or a product on Amazon, remember that your request isn't simply matching text. It's being interpreted, expanded, ranked, personalized, and optimized through dozens of intelligent systems working together behind the scenes.

That tiny search box represents years of engineering innovation—and it's one of the best examples of how data structures, algorithms, AI, and software engineering come together to create an experience that feels almost effortless.

Gradient Descent: The Engine That Made Deep Learning Possible : How one simple idea changed the way machines learn

Vineet Chauhan — Fri, 03 Jul 2026 14:45:02 +0000

When I first started learning deep learning, I thought the magic was inside the model architecture.

CNNs looked powerful.

RNNs looked intelligent.

Transformers looked almost impossible to understand.

But slowly I realized something important.

Architecture is only one part of deep learning.

The real question is:

How does the model actually learn?

A neural network may contain millions or even billions of parameters.

But at the beginning, all those parameters are almost useless.

They are usually random numbers.

The model does not know anything.

It does not understand images.

It does not understand language.

It does not understand patterns.

So the real magic is not that a neural network has many parameters.

The real magic is that it can adjust those parameters automatically.

That automatic adjustment is made possible by Gradient Descent.

The Basic Problem

Suppose we are training a simple model.

y_pred = w * x + b

Here:

w = weight
b = bias

At first, the model makes wrong predictions.

So we calculate error.

loss = (y_actual - y_pred) ** 2

The goal is simple:

Reduce the loss

But the question is:

How should w and b change?

Should weight increase?

Should weight decrease?

By how much?

This is where Gradient Descent comes in.

What Gradient Descent Really Means

Gradient Descent simply means:

Move the parameters in the direction where loss decreases.

Imagine standing on a mountain in fog.

You cannot see the full path.

You only know the slope under your feet.

So you move downward step by step.

That is Gradient Descent.

In machine learning:

Mountain height = Loss
Position = Parameters
Downward direction = Negative gradient

The formula is:

new_weight = old_weight - learning_rate × gradient

or:

w = w - lr * dw
b = b - lr * db

This small formula is one of the biggest reasons deep learning works.

A Very Small Numerical Example

Suppose our model is:

y_pred = w * x

Let:

x = 2
y_actual = 10
w = 1

Prediction:

y_pred = 1 × 2 = 2

Loss:

loss = (10 - 2)² = 64

The prediction is too small.

So weight should increase.

Gradient Descent tells us exactly how much to update.

For squared error:

loss = (y - wx)²

Gradient with respect to weight:

dL/dw = -2x(y - wx)

Now substitute values:

dL/dw = -2 × 2 × (10 - 2)
      = -4 × 8
      = -32

Let learning rate be:

lr = 0.1

Update:

new_w = old_w - lr × gradient
new_w = 1 - 0.1 × (-32)
new_w = 1 + 3.2
new_w = 4.2

Now prediction becomes:

y_pred = 4.2 × 2 = 8.4

Earlier prediction was 2.

Now it is 8.4.

Much closer to 10.

That is learning.

Not memorization.

Not magic.

Just repeated improvement.

Code: Gradient Descent From Scratch

import numpy as np
import matplotlib.pyplot as plt

x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 6, 8, 10])

w = 0
b = 0

lr = 0.01
epochs = 1000

losses = []

for epoch in range(epochs):
    y_pred = w * x + b

    loss = np.mean((y - y_pred) ** 2)
    losses.append(loss)

    dw = (-2 / len(x)) * np.sum(x * (y - y_pred))
    db = (-2 / len(x)) * np.sum(y - y_pred)

    w = w - lr * dw
    b = b - lr * db

print("Final weight:", w)
print("Final bias:", b)

Expected output:

Final weight: close to 2
Final bias: close to 0

The model discovers:

y = 2x

by updating weights again and again.

Plotting The Loss Curve

plt.plot(losses)
plt.xlabel("Epochs")
plt.ylabel("Loss")
plt.title("Loss Decreasing During Gradient Descent")
plt.show()

This graph is very important.

If loss goes down, the model is learning.

If loss goes up, learning rate may be too high.

If loss is flat, learning rate may be too low or model may not be powerful enough.

What Happens If Learning Rate Is Too High?

Suppose learning rate is very large.

Then the model takes huge jumps.

Instead of reaching the minimum, it may jump over it again and again.

Good learning rate:

Loss → ↓ ↓ ↓ ↓ ↓

Too high learning rate:

Loss → ↑ ↓ ↑ ↓ ↑

Code experiment:

lr = 1.0

You may see loss explode.

This is called divergence.

What Happens If Learning Rate Is Too Low?

If learning rate is too small:

lr = 0.000001

The model learns extremely slowly.

Loss decreases, but almost nothing happens for many epochs.

This is why learning rate is one of the most important hyperparameters in deep learning.

Why Gradient Descent Changed Deep Learning

Before deep learning became powerful, one big problem was:

How do we train huge models?

A deep neural network may have:

Millions of weights
Millions of biases
Multiple layers
Complex activations
Huge datasets

Manually choosing weights is impossible.

Trying all combinations is impossible.

Gradient Descent made training possible because it gave a systematic way to improve every parameter.

Even if a model has 10 million parameters, the idea remains:

Find gradient
Move opposite to gradient
Reduce loss
Repeat

That is why Gradient Descent became the engine of deep learning.

What Happens If There Is No Gradient Descent?

Without Gradient Descent, deep learning would almost collapse.

We would have neural networks, but we would not know how to train them efficiently.

Without Gradient Descent:

No automatic weight improvement
No large-scale neural network training
No modern computer vision
No powerful language models
No practical deep learning revolution

We could still use some alternatives like:

Random search
Genetic algorithms
Manual tuning
Closed-form solutions for very small models

But they do not scale like Gradient Descent.

Imagine a neural network with 100 million parameters.

Randomly trying weights would be hopeless.

Gradient Descent gives direction.

That direction changed everything.

Gradient Descent In Neural Networks

In a neural network, every layer has weights.

Input → Hidden Layer → Output

Each layer makes a small transformation.

The final prediction produces loss.

Then backpropagation calculates:

How much each weight contributed to the error

Gradient Descent then updates all weights.

optimizer.zero_grad()
loss.backward()
optimizer.step()

This PyTorch code looks small.

But conceptually:

loss.backward()

calculates gradients.

optimizer.step()

applies Gradient Descent.

That is the heart of deep learning training.

PyTorch Example

import torch
import torch.nn as nn

X = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
y = torch.tensor([[2.0], [4.0], [6.0], [8.0]])

model = nn.Linear(1, 1)

loss_fn = nn.MSELoss()

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for epoch in range(1000):
    y_pred = model(X)

    loss = loss_fn(y_pred, y)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(list(model.parameters()))

This model also learns:

y = 2x

The difference is that PyTorch calculates gradients automatically.

That automatic gradient calculation is called autograd.

Gradient Descent vs Backpropagation

Many beginners confuse these two.

They are related but not the same.

Backpropagation answers:

What are the gradients?

Gradient Descent answers:

How should we update the weights using those gradients?

So:

Backpropagation = gradient calculation

Gradient Descent = parameter update

Together, they train deep neural networks.

Why Deep Learning Needed Gradient Descent More Than Traditional ML

Traditional ML models often have fewer parameters.

Some algorithms do not rely heavily on gradients.

For example:

Decision Trees split data using rules.

KNN stores examples.

Naive Bayes uses probability formulas.

But deep learning is different.

Deep learning is mostly parameter learning.

Millions of parameters must be adjusted.

That is why Gradient Descent became more important in deep learning than almost anywhere else.

Final Thought

Gradient Descent changed deep learning because it converted learning into optimization.

Instead of manually programming intelligence, we define:

Model
Loss Function
Optimizer
Data

Then the model improves itself step by step.

That is the real breakthrough.

Deep learning is not just about big neural networks.

It is about trainable neural networks.

And Gradient Descent is what makes them trainable.

Without it, deep learning would be like a powerful engine with no steering.

With it, random weights slowly become useful knowledge.

Deep Learning Is More Logistic Regression Than You Think

Vineet Chauhan — Sat, 06 Jun 2026 20:01:20 +0000

Why an Algorithm From the 1950s Still Powers Modern AI

When I first learned Machine Learning, I treated Logistic Regression as a beginner algorithm.

You learn it.

You build a classifier.

You get an accuracy score.

Then you move on.

At least that's what I thought.

After Logistic Regression came:

Decision Trees
Random Forests
XGBoost
Neural Networks
Transformers
Large Language Models

The journey seemed straightforward.

Old algorithm → Better algorithm → Even better algorithm.

But after studying Deep Learning more to some extent, I discovered something surprising.

The algorithm I thought I had left behind was everywhere.

Not Decision Trees.

Not Random Forests.

Not SVMs.

Logistic Regression.

And the deeper I looked, the more I realized that modern Deep Learning did not replace Logistic Regression.

It scaled its ideas.

The First Time I Noticed It

I was learning about neural networks.

The instructor drew a neuron:

z = w1*x1 + w2*x2 + b
output = sigmoid(z)

I stared at the equation for a few seconds.

Then it hit me.

That is literally Logistic Regression.

The exact same weighted sum.

The exact same sigmoid activation.

The exact same probability output.

The exact same optimization process.

The only difference?

A neural network has many of them.

What Exactly Does Logistic Regression Do?

At its core, Logistic Regression performs two operations.

Step 1:

Take a weighted sum.

z = w1*x1 + w2*x2 + w3*x3 + b

Step 2:

Convert it into probability.

sigmoid(z) = 1/1+e^-x

The sigmoid function transforms any number into a value between 0 and 1.

Example:

Input = -10 → 0.00004

Input = 0 → 0.5

Input = 10 → 0.99995

This probability becomes the final prediction.

Simple.

Elegant.

Effective.

Now Look At A Neural Network

A neuron performs:

z = w1*x1 + w2*x2 + b

Then:

output = activation(z)

In early neural networks:

activation = sigmoid

which means:

Neuron
=
Logistic Regression Unit

The moment I realized this, neural networks became much easier to understand.

Instead of imagining some magical AI machine, I started seeing thousands of Logistic Regression models stacked together.

Why Not Decision Trees?

This question bothered me for a long time.

Why didn't Deep Learning evolve from Decision Trees?

Why not Random Forests?

Why specifically Logistic Regression?

The answer lies in mathematics.

Reason 1: Logistic Regression Is Differentiable

Decision Trees make hard decisions.

Age > 30 ?

Yes → Left

No → Right

A tiny change in age can suddenly change the entire path.

This creates discontinuities.

Gradient Descent cannot work efficiently.

Logistic Regression is different.

Its sigmoid curve is smooth.

Every tiny change produces a tiny output change.

This makes gradients possible.

And gradients are the fuel of Deep Learning.

Without gradients:

No Backpropagation

Without backpropagation:

No Neural Networks

Without neural networks:

No ChatGPT

Reason 2: Logistic Regression Produces Probabilities

A Decision Tree says:

Class A

Class B

Logistic Regression says:

P(Class A) = 0.92

Probability matters.

Modern AI relies heavily on probabilities.

Examples:

Spam Detection

98% Spam

Medical Diagnosis

73% Cancer Risk

Language Models

P(next word = "cat")

Transformers are fundamentally probability machines.

And Logistic Regression introduced that philosophy long ago.

Reason 3: Cross Entropy Came From Logistic Regression

One of the most important loss functions in Deep Learning is:

L=-[y\log(p)+(1-y)\log(1-p)]

Almost every deep learning engineer uses it.

Image Classification.

Medical AI.

Fraud Detection.

NLP.

Large Language Models.

The interesting part?

This is the same loss function used in Logistic Regression.

The entire deep learning world still depends on it.

Reason 4: Logistic Regression Is A Single Neuron

This was the biggest realization for me.

A Logistic Regression model can be represented as:

Input
 ↓
Weighted Sum
 ↓
Sigmoid
 ↓
Output

Now look at a neural network:

Input
 ↓
100 Neurons
 ↓
100 Neurons
 ↓
100 Neurons
 ↓
Output

Each neuron is doing a very similar operation.

The network becomes powerful because thousands of these simple units collaborate.

Deep Learning is not complexity replacing simplicity.

It is simplicity repeated at scale.

Let's Verify This With Code

Logistic Regression

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()

model.fit(X_train, y_train)

predictions = model.predict(X_test)

Now a neural network.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10,1),
    nn.Sigmoid()
)

Look carefully.

Both perform:

Weighted Sum
Sigmoid Transformation
Probability Prediction

Mathematically they are nearly identical.

The PyTorch version is essentially Logistic Regression implemented as a neural network.

The Real Difference

If they are so similar, why use Deep Learning?

Because Logistic Regression can only learn simple boundaries.

Imagine separating red and blue dots.

Logistic Regression creates:

One Straight Line

Deep Learning creates:

Curves
Shapes
Complex Regions
Non-Linear Patterns

By stacking layers, the network gradually transforms simple linear boundaries into highly complex decision surfaces.

That is the true power of Deep Learning.

Not a different idea.

A larger version of the same idea.

The Most Surprising Place I Found Logistic Regression

LSTMs.

The architecture behind many sequence models.

Inside every LSTM cell are gates.

Forget Gate.

Input Gate.

Output Gate.

Guess what activation function they use?

Sigmoid.

Every gate computes probabilities.

Every gate decides:

Keep Information?

Forget Information?

using Logistic Regression principles.

Even modern AI systems still carry its DNA.

Final Thoughts

When I first learned Logistic Regression, I thought it was something to finish and forget.

Now I see it differently.

I see it as the first neural network.

I see it as the origin of probability-based learning.

I see it as the mathematical foundation behind cross entropy, gradient descent, and backpropagation.

The next time someone says Logistic Regression is an old algorithm, remember:

Deep Learning did not replace Logistic Regression.

Deep Learning scaled it.

And some of the most advanced AI systems ever built still rely on ideas introduced by Logistic Regression decades ago.

Better Data Beats Better Algorithms: Before Changing the Model, Change the Data

Vineet Chauhan — Sat, 06 Jun 2026 19:47:54 +0000

How Feature Engineering Taught Me That Better Data Often Beats Better Algorithms

When I first started learning Machine Learning, I believed what many beginners believe:

If my model is not performing well, I need a better algorithm.

So I kept switching models.

I moved from Logistic Regression to Decision Trees, then Random Forest, and later even started reading about XGBoost and Neural Networks.

The results improved slightly, but never dramatically.

What surprised me was that the biggest improvement didn't come from changing the algorithm.

It came from changing the data.

The Problem

I was working on a dataset containing missing values, outliers, and categorical variables.

Like many beginners, my first instinct was simple:

model.fit(X_train, y_train)
pred = model.predict(X_test)

The model trained successfully.

The accuracy looked acceptable.

But something felt wrong.

The data itself was messy.

Some columns contained missing values.

Some numerical features had extreme outliers.

Several categorical columns were represented as text.

Yet I expected the model to magically learn everything.

My First Experiment

I trained a Logistic Regression model on the raw dataset.

Results:

Accuracy : 72%

Not terrible.

Not impressive either.

Instead of changing the model, I decided to investigate the data.

This turned out to be the most important decision of the entire project.

Step 1: Handling Missing Values

The dataset contained several missing values.

At first I considered simply deleting rows.

df.dropna(inplace=True)

The problem?

I lost a significant portion of the data.

So I experimented with multiple approaches:

Mean Imputation

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy='mean')
X = imputer.fit_transform(X)

Median Imputation

imputer = SimpleImputer(strategy='median')

KNN Imputation

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)
X = imputer.fit_transform(X)

KNN preserved relationships between records much better than simple averaging.

This alone improved performance.

Step 2: Fighting Outliers

I then visualized the numerical columns.

The boxplots looked terrible.

A few extreme values were stretching entire distributions.

sns.boxplot(df["experience"])

The model was spending too much effort trying to fit a handful of unusual observations.

I used IQR-based treatment.

Q1 = df["experience"].quantile(0.25)
Q3 = df["experience"].quantile(0.75)

IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

df = df[(df["experience"] >= lower) &
        (df["experience"] <= upper)]

After removing outliers, the data distribution became much cleaner.

More importantly, the model began learning actual patterns instead of noise.

Step 3: Encoding Categorical Features

Machine Learning algorithms cannot understand text.

They only understand numbers.

So columns like:

Male
Female

Private
Public

Graduate
Masters

needed transformation.

I applied One-Hot Encoding.

pd.get_dummies(df,
               columns=["gender",
                        "company_type"])

and Ordinal Encoding where order mattered.

education_level

High School
Graduate
Masters
PhD

This converted human-readable categories into machine-readable information.

Step 4: Feature Scaling

Some columns ranged between:

0 – 5

while others ranged between:

0 – 100000

Distance-based algorithms become biased toward larger values.

I applied MinMax Scaling.

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Now every feature contributed fairly.

What Happened Next?

I trained the exact same Logistic Regression model again.

Nothing changed except the data.

Results:

Before Feature Engineering : 72%

After Feature Engineering  : 86%

A gain of 14 percentage points.

Without changing the algorithm.

Without using deep learning.

Without adding complexity.

Just by improving the data.

The Most Important Lesson

This project changed the way I think about Machine Learning.

Earlier I believed:

Better Algorithm
       ↓
Better Results

Now I believe:

Better Data
       ↓
Better Features
       ↓
Better Results

Most real-world machine learning problems are not algorithm problems.

They are data problems.

A powerful model trained on poor-quality data will still struggle.

A simple model trained on clean, meaningful data can often outperform much more complex alternatives.

Challenges I Faced

The hardest part was not training the model.

The hardest part was preparing the data.

Some difficulties included:

Losing rows during Complete Case Analysis
Choosing between Mean, Median, and KNN Imputation
Combining transformed datasets
Handling dimensionality after One-Hot Encoding
Identifying genuine outliers versus valuable rare cases

These challenges taught me more than model training ever did.

Final Thoughts

Feature Engineering is not the most glamorous part of Machine Learning.

Nobody posts screenshots of missing value treatment on social media.

Nobody celebrates scaling features.

Yet this is where much of the real improvement happens.

After this project, I stopped asking:

Which model should I use?

and started asking:

What is my data trying to tell me?

That single change in mindset improved my machine learning skills more than learning any new algorithm.

Feature Engineering is Not Just “Cleaning Data”: What I Learned While Building a Real ML Pipeline

Vineet Chauhan — Thu, 28 May 2026 10:43:45 +0000

Most machine learning tutorials make preprocessing look straightforward.

Handle missing values.
Encode categorical features.
Train the model.
Get accuracy.

But while working on a real classification dataset, I realized feature engineering is far less about applying textbook techniques and far more about making careful decisions under uncertainty.

This project completely changed how I think about preprocessing.

Instead of writing another “complete guide to feature engineering,” I wanted to document the actual engineering problems I faced while building a preprocessing pipeline — including debugging mistakes, failed assumptions, distribution shifts, encoding challenges, and how preprocessing itself changed model behaviour.

The Dataset Looked Simple at First

Initially, the dataset looked manageable:

Numerical features
Categorical features
Missing values
Binary target variable

Nothing seemed unusual.

But the moment preprocessing started, the real complexity appeared.

The First Problem: Missing Values Were Uneven Everywhere

One of the first things I checked was the percentage of missing values across columns.

Some columns had:

less than 1% missing values
some had 5–10%
others had more than 30%

This immediately raised an important question:

Should every missing value be handled using the same strategy?

The answer quickly became no.

My Initial Mistake: Applying One Strategy Everywhere

At first, I tried treating all missing values similarly.

That approach failed quickly because:

Complete Case Analysis removed too many rows
KNN Imputation behaved poorly on categorical-heavy features
Encoded categorical values introduced unrealistic numeric relationships
Feature distributions started changing unexpectedly

This was the first moment I realized:

Feature engineering is not a fixed recipe.

Different features require different preprocessing decisions.

Using Complete Case Analysis (CCA)

For columns with less than 5% missing values, I used Complete Case Analysis.

At first, this seemed harmless.

But then I decided to compare feature distributions before and after row deletion.

This turned out to be one of the most important observations in the project.

Even small row deletions slightly changed feature density and distributions.

That was the moment I understood:

Missing value handling is not only about removing NaNs — it can also reshape the dataset itself.

A Small Pandas Mistake That Broke My Pipeline

One debugging issue confused me for quite a while.

Initially, I wrote:

df[cols].dropna()

This unintentionally removed all other columns from the dataframe.

The correct approach was:

df.dropna(subset=cols)

The difference was tiny syntactically but huge logically.

This taught me something surprisingly important:

Many machine learning problems are not model problems.
They are dataframe manipulation problems.

Why KNN Imputer Became Complicated

Initially, I planned to use KNN Imputer for all remaining missing values.

But another issue appeared immediately.

KNN relies on distance calculations.

Distance works naturally for numerical data, but categorical columns require encoding first.

That introduced several complications:

Label encoding created artificial numeric relationships
One Hot Encoding exploded feature dimensionality
NaN values converted into strings accidentally during preprocessing
Encoded categories distorted neighbor similarity

This made me realize:

KNN Imputer works much better for numerical features than heavily categorical datasets.

Eventually, I switched to a hybrid preprocessing strategy instead of forcing one solution everywhere.

Final Missing Value Strategy

I ended up using:

Feature Type:- Strategy:-

Low missing numerical ---> Complete Case Analysis
High missing numerical ---> Median/KNN Imputation
High missing categorical ---> Most Frequent / “Missing” category

This hybrid approach worked far better than blindly applying one technique globally.

Encoding Was More Important Than I Expected

Encoding looked simple in theory.

But in practice, deciding between:

One Hot Encoding
Ordinal Encoding
Label Encoding

actually mattered a lot.

I used:

One Hot Encoding

For:

gender
major_discipline
company_type
enrolled_university

because these categories had no natural order.

Ordinal Encoding

For:

education_level

because educational levels actually contain ranking:

Primary School < High School < Graduate < Masters < PhD

This distinction improved model behavior more than I initially expected.

The Most Interesting Observation: Preprocessing Changed the Models More Than the Models Changed Themselves

I trained multiple models:

Logistic Regression
Decision Tree
Random Forest

both before and after preprocessing.

The results were surprisingly different.

Linear models improved heavily after scaling and proper encoding.

Random Forest remained comparatively stable even before aggressive preprocessing.

That observation completely changed my perspective.

Data preprocessing often influences performance more than changing the algorithm itself.

Another Real Problem: Feature Explosion

After applying One Hot Encoding, the number of features increased rapidly.

This was another practical challenge rarely discussed in beginner tutorials.

Encoding solved categorical representation issues, but it also increased dimensionality and preprocessing complexity significantly.

Why Pipelines Became Necessary

At one point, preprocessing became chaotic.

Different transformations were happening separately:

scaling
imputation
encoding
train-test transformations

Tracking transformed columns manually became painful.

This was when I finally understood why sklearn Pipelines and ColumnTransformers matter so much.

Not because they look advanced —
but because preprocessing becomes unmanageable very quickly in real projects.

What This Project Changed for Me

Before this project, I thought feature engineering mostly meant:

removing null values
encoding categories
scaling features

Now I think differently.

Feature engineering is closer to:

understanding how data behaves under transformation.

Every preprocessing decision changes:

distributions
feature relationships
dimensionality
information retention
model assumptions

Even small preprocessing choices can significantly change model behavior.

*Final Thoughts
*
One thing became very clear after this project:

Machine learning is not just model training.

Most real effort goes into:

understanding data
debugging preprocessing
handling edge cases
preserving useful information
testing assumptions

Feature engineering is where datasets stop behaving like clean classroom examples and start behaving like real systems.

"And honestly, that is where machine learning starts becoming interesting."

Building a Hantavirus Misinformation Detector: Challenges of NLP in Low-Data Health Domains

Vineet Chauhan — Sat, 16 May 2026 17:00:47 +0000

Most fake news detection projects rely on massive datasets containing thousands of examples.

I wanted to explore something much more difficult:

Can a small NLP system detect misinformation around an emerging disease like Hantavirus?

What made this project interesting was not the model itself, but the challenge of working in a low-data environment where reliable misinformation examples barely exist.

Unlike COVID-19 misinformation datasets, hantavirus-related misinformation is extremely limited online. This forced me to manually curate both factual and misleading claims while understanding how health misinformation behaves linguistically.

This project became less about achieving high accuracy and more about:

understanding NLP pipelines,
handling imperfect datasets,
and analyzing misinformation patterns.

2. Understanding the Problem

Health misinformation spreads differently from normal fake news.

Many misleading claims are:

1. partially believable,
2. emotionally framed,
3. or based on incomplete truths.

Examples:

“Natural remedies can cure hantavirus”
“Governments are hiding outbreak data”
“Hot water prevents infection”

The challenge was not simply classifying text as fake or real, but understanding how subtle misinformation patterns emerge in health-related discussions.

3. Dataset Creation (The Hardest Part)

This was by far the most difficult stage of the project.

Unlike mainstream misinformation domains, there are very few structured datasets specifically related to hantavirus misinformation. Because of this, I manually curated a small dataset using:

trusted medical sources,
news articles,
and realistic misinformation patterns.

4. Real Data Sources

I collected factual information from:

WHO
CDC
Reuters

Examples included:

transmission details,
symptoms,
prevention methods,
and treatment limitations.

5. Fake Data Construction

Finding real misinformation examples for hantavirus was difficult because the topic is relatively niche.

Instead of generating random false statements, I focused on realistic misinformation patterns commonly seen in health-related fake news:

miracle cures,
conspiracy theories,
exaggerated transmission claims,
and misleading prevention methods.

Examples:

“Garlic water can completely cure hantavirus”
“The virus spreads rapidly through city air systems”
“A secret vaccine already exists”

Dataset Structure

The dataset included:

text
label
source
category
difficulty

This structure helped organize misinformation types and analyze which claims were easier or harder for the model to classify.

7. Dataset Analysis

Fake vs Real Distribution

Category Distribution

Difficulty Distribution

8. NLP Pipeline

The NLP pipeline was intentionally kept simple to better understand the fundamentals.

The workflow consisted of:

Text preprocessing
TF-IDF vectorization
Logistic Regression classification

9. Text Preprocessing

The first step involved cleaning the text data:

converting text to lowercase,
removing punctuation,
removing unnecessary spaces,
and standardizing sentence structure.

10. TF-IDF Vectorization

Machine learning models cannot directly understand raw text.

TF-IDF converts words into numerical representations based on their importance across the dataset.

This allowed the model to identify patterns such as:

“secret cure”
“government hiding”
“supportive care”
“WHO reports”

11. Most Interesting Observation

One of the most surprising findings was:

believable misinformation is much harder to classify than extreme misinformation.

Claims like:

“Herbal remedies may reduce hantavirus symptoms”

were more difficult for the model than clearly absurd claims.

This highlighted an important limitation of simple NLP models:
they rely heavily on statistical language patterns rather than true medical understanding.

12. Limitations

This project has several limitations:

small dataset size,
manually curated misinformation,
limited real-world social media data,
and no deep learning models.

Because of these constraints, the model should not be treated as a production-ready misinformation detector.

Instead, this project should be viewed as:

an exploratory NLP experiment in a low-data health misinformation domain.

13. Future Improvements

There are several directions for improving this project:

collecting real social media misinformation,
increasing dataset size,
using transformer-based models like BERT,
multilingual misinformation detection,
and explainable AI methods such as SHAP or LIME.

14. Final Thoughts

This project taught me that the hardest part of NLP is often not the model itself.

It is:

collecting meaningful data,
understanding ambiguity,
and dealing with imperfect real-world information.

Working on a low-data problem like hantavirus misinformation made the project far more challenging — and far more educational — than simply training a model on a large public dataset.

Even though the model itself was simple, the process revealed how difficult health misinformation detection actually is in practice