How I applied Exploratory Data Analysis, Feature Engineering, Pipelines, and Ensemble Models to solve a real-world machine learning problem—and the lessons I learned along the way.
Introduction
There comes a point in every machine learning learner's journey when watching tutorials and completing small practice exercises are no longer enough.
After spending weeks understanding statistics, exploratory data analysis (EDA), feature engineering, preprocessing techniques, and classical machine learning algorithms, I wanted to answer one question:
Can I apply everything I've learned to a real machine learning competition?
That's when I decided to participate in a Kaggle Playground competition.
Unlike classroom datasets, Kaggle competitions force you to think like a machine learning engineer. You're responsible for understanding messy data, building preprocessing pipelines, selecting models, evaluating performance, debugging errors, and finally creating a submission that competes with thousands of participants.
This article documents my complete journey—from loading the dataset to building production-style preprocessing pipelines and training multiple ensemble models. Along the way, I'll also share the challenges I faced, what worked well, and the lessons I'll carry into future competitions.
Why Kaggle?
Learning machine learning isn't just about knowing algorithms.
Real-world ML requires answering questions like:
- Which features are useful?
- How should missing values be handled?
- Should categorical variables be one-hot encoded or ordinal encoded?
- Which preprocessing steps belong inside a pipeline?
- How do different ensemble models compare?
Kaggle provides an environment where all of these questions matter.
Instead of building a model that works only inside a notebook, you're solving a problem under realistic constraints and evaluating your solution on unseen data.
Competition Goal
The objective of this Playground competition was to predict the target class based on a combination of numerical and categorical features related to health and lifestyle.
The workflow followed the same structure used in many real-world machine learning projects.
Raw Dataset
│
▼
Exploratory Data Analysis
│
▼
Missing Value Analysis
│
▼
Feature Engineering
│
▼
Preprocessing Pipeline
│
▼
Model Training
│
▼
Evaluation
│
▼
Prediction
│
▼
Kaggle Submission
Although the workflow appears simple, every stage requires careful decision-making.
Understanding the Dataset
The first step was loading both the training and testing datasets using Pandas.
train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")
Before writing a single machine learning model, I spent time understanding the structure of the data.
Some of the questions I explored were:
- How many rows and columns are present?
- Which features are numerical?
- Which features are categorical?
- Are there missing values?
- What is the distribution of each feature?
- Are there any obvious outliers?
Skipping this stage often leads to poor model performance because preprocessing decisions depend entirely on the characteristics of the data.
Exploratory Data Analysis (EDA)
EDA turned out to be one of the most valuable parts of the project.
Instead of immediately training a model, I wanted to understand how the dataset behaved.
I separated the features into two broad categories:
Numerical Features
Examples included:
- Sleep Duration
- Heart Rate
- BMI
- Calorie Expenditure
- Step Count
- Exercise Duration
- Water Intake
Categorical Features
Examples included:
- Diet Type
- Gender
- Smoking/Alcohol
- Physical Activity Level
- Sleep Quality
- Stress Level
This separation made it much easier to apply different visualization techniques.
Visualizing Numerical Features
For numerical columns, I explored the data using:
- Histograms
- KDE Plots
- Boxplots
These visualizations helped answer important questions.
Are the features normally distributed?
Some variables showed distributions close to normal, while others exhibited noticeable skewness.
Are there extreme values?
Boxplots revealed the presence of outliers in several numerical features.
Is the data symmetric?
Using skewness calculations together with KDE plots made it easier to understand whether transformations might be useful.
Instead of blindly preprocessing the data, these visualizations allowed me to make decisions based on evidence.
Understanding Missing Values
One of the first analyses I performed was checking the percentage of missing values across the dataset.
train.isnull().mean() * 100
Missing values are one of the most common problems in real-world datasets.
Ignoring them isn't an option because most machine learning algorithms cannot train with incomplete data.
Rather than replacing every missing value with a single constant, I decided to use appropriate preprocessing techniques later in the pipeline.
Exploring Categorical Features
For categorical variables, I created count plots to understand the frequency distribution of each category.
These plots answered questions like:
- Which diet type appears most frequently?
- How are physical activity levels distributed?
- Does gender influence certain lifestyle variables?
- Are some categories extremely rare?
I also explored relationships between different categorical variables using grouped visualizations.
This helped me develop an intuition about the dataset before training any model.
Building Intuition Before Building Models
One lesson became very clear during EDA:
Better understanding often leads to better preprocessing.
Machine learning isn't just about choosing a powerful algorithm.
The quality of the input data has an enormous impact on the quality of the final predictions.
That's why I spent significant time exploring the data before moving on to feature engineering.
Feature Engineering
After understanding the dataset, the next step was preparing it for machine learning.
The first task was separating features and the target variable.
X = train.drop(columns=["health_condition"])
y = train["health_condition"]
This created a clear distinction between:
- Input features (
X) - Target labels (
y)
I then split the training data into training and validation sets.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42
)
Creating a validation set allowed me to evaluate model performance before making a Kaggle submission.
Why Preprocessing Matters
Different feature types require different preprocessing strategies.
For example:
| Feature Type | Preprocessing |
|---|---|
| Numerical | Imputation + Scaling |
| Ordinal Categorical | Ordinal Encoding |
| Nominal Categorical | One-Hot Encoding |
| Target Variable | Label Encoding |
Applying the wrong preprocessing technique can significantly reduce model performance.
Instead of manually transforming every column, I wanted a cleaner and more reusable solution.
That's where Scikit-learn Pipelines became extremely useful.
Organizing Features
I divided the features into logical groups.
Numerical Columns
Continuous numerical variables requiring scaling and missing value handling.
Ordinal Categories
Features with a natural ordering, such as activity or stress levels.
These can be safely converted into ordered numerical values.
Nominal Categories
Features such as gender or diet type have no natural ordering.
Assigning arbitrary numbers could introduce unintended relationships, so One-Hot Encoding is a better choice.
This separation made the preprocessing pipeline much easier to maintain.
Building a Production-Style Preprocessing Pipeline
Rather than writing preprocessing code repeatedly for every model, I used Scikit-learn's Pipeline and ColumnTransformer.
This approach provides several advantages:
- Cleaner code
- Reduced risk of data leakage
- Consistent preprocessing
- Easier experimentation with multiple models
- Production-ready workflow
Instead of manually applying transformations one by one, the pipeline automatically performs every preprocessing step before model training.
This is one of the biggest improvements I made compared to my earlier machine learning projects.
Building a Production-Ready Machine Learning Pipeline
One of the biggest improvements I made in this project compared to my earlier machine learning exercises was moving away from manual preprocessing.
Instead of writing separate preprocessing code for every model, I built a reusable pipeline using Scikit-learn.
This approach follows the same philosophy used in production machine learning systems.
Rather than remembering dozens of preprocessing steps, everything is automated inside a single workflow.
Designing the Preprocessing Pipeline
Different feature types require different transformations.
My preprocessing workflow looked like this:
Raw Dataset
│
▼
Separate Numerical & Categorical Features
│
├───────────────┐
│ │
▼ ▼
Numerical Categorical
│ │
KNN Imputer Simple Imputer
│ │
StandardScaler Encoding
│ │
└──────┬────────┘
▼
ColumnTransformer
▼
Machine Learning Model
Instead of transforming every column manually, the pipeline automatically applied the correct preprocessing to each feature type.
This made the code cleaner, reusable, and less error-prone.
Handling Missing Values
Missing values are unavoidable in real-world datasets.
Instead of dropping rows and losing valuable information, I applied different imputation strategies depending on the feature type.
For numerical features, I used KNN Imputer.
The idea behind KNN Imputation is simple.
Instead of filling missing values with a mean or median, it looks for the most similar observations and estimates the missing value based on those neighbors.
For categorical features, I used Simple Imputer with the most frequent category.
This preserved the integrity of categorical data while avoiding unnecessary data loss.
Encoding Categorical Variables
Machine learning models cannot understand text directly.
Therefore, categorical variables needed to be converted into numerical representations.
I divided categorical columns into two groups.
Ordinal Features
These have a natural order.
Examples include activity levels or stress levels.
For these variables, I used Ordinal Encoding.
Low → 0
Medium → 1
High → 2
Since these categories have meaningful ordering, ordinal encoding preserves that relationship.
Nominal Features
Some features have no natural ordering.
Examples include gender or diet type.
Assigning numbers such as
Male = 0
Female = 1
could unintentionally imply a mathematical relationship.
Instead, I applied One-Hot Encoding, where each category receives its own binary column.
This prevents the model from assuming nonexistent order.
Why I Used ColumnTransformer
Without a ColumnTransformer, preprocessing quickly becomes difficult to manage.
Different feature groups require different transformations.
Instead of manually applying each transformation, ColumnTransformer allows everything to happen in one unified preprocessing stage.
This makes experimentation much easier because changing the model no longer requires rewriting preprocessing code.
Building Reusable Pipelines
After creating the preprocessing pipeline, I combined it with different machine learning algorithms.
Instead of writing separate preprocessing code for every model, I simply replaced the final estimator.
Conceptually, every pipeline followed this structure:
Preprocessing
│
▼
Machine Learning Model
│
▼
Predictions
This allowed me to compare multiple algorithms while keeping preprocessing completely consistent.
Training Multiple Ensemble Models
Rather than relying on a single algorithm, I trained several ensemble models to compare their performance.
The models included:
- Random Forest
- XGBoost
- CatBoost
- LightGBM
Each model has its own strengths.
Random Forest
Random Forest was my baseline ensemble model.
Its advantages include:
- Robust performance
- Handles nonlinear relationships
- Resistant to overfitting
- Useful for feature importance
It gave me a strong benchmark before experimenting with boosting algorithms.
XGBoost
XGBoost is one of the most popular gradient boosting libraries in competitive machine learning.
In my implementation, I configured parameters such as:
- 300 estimators
- Learning rate of 0.05
- Maximum tree depth
- Subsampling
- Column sampling
- Histogram-based tree construction
- GPU acceleration
The goal was to balance predictive performance with computational efficiency.
CatBoost
CatBoost is particularly effective when working with datasets containing categorical variables.
Even though my preprocessing pipeline already handled encoding, CatBoost still provided another strong ensemble model for comparison.
I trained it using GPU acceleration to reduce computation time.
LightGBM
LightGBM is designed for efficiency.
It uses histogram-based learning and leaf-wise tree growth, making it extremely fast while maintaining competitive performance.
Its ability to scale efficiently makes it a popular choice in Kaggle competitions.
Evaluating Model Performance
Training a model is only half of the workflow.
The next step is evaluating how well it performs.
I compared the models using several evaluation metrics instead of relying on accuracy alone.
These included:
- Accuracy
- Precision
- Recall
- F1 Score
Each metric provides different insights into model behavior.
Looking at multiple metrics helps avoid misleading conclusions that can arise from using only accuracy.
Classification Reports
To gain a deeper understanding of performance, I generated classification reports for each model.
These reports show:
- Precision for every class
- Recall for every class
- F1-score
- Overall accuracy
- Macro average
- Weighted average
Instead of treating the model as a black box, these reports reveal where it performs well and where it struggles.
Confusion Matrix
Another valuable visualization was the confusion matrix.
Rather than simply knowing that a prediction was incorrect, the confusion matrix shows which classes the model confuses with one another.
This often reveals patterns that raw accuracy cannot.
For multiclass classification problems, confusion matrices provide valuable insights into model weaknesses.
Comparing Models
Instead of evaluating models separately, I created a comparison table containing metrics such as:
| Model | Accuracy | Precision | Recall | F1 Score |
|---|---|---|---|---|
| XGBoost | Compared | Compared | Compared | Compared |
| CatBoost | Compared | Compared | Compared | Compared |
| LightGBM | Compared | Compared | Compared | Compared |
A comparison table makes it much easier to identify trade-offs between different algorithms.
Lessons I Learned
Looking back, this competition taught me far more than simply training machine learning models.
Some of the most important lessons were:
Understanding the data matters more than choosing the model.
A strong preprocessing pipeline often improves performance more than switching algorithms.
Pipelines save enormous amounts of time.
Instead of rewriting preprocessing code repeatedly, everything becomes modular and reusable.
Evaluation should never rely on a single metric.
Accuracy alone rarely tells the complete story.
Ensemble methods are powerful.
Comparing multiple algorithms helped me understand the strengths and limitations of each approach.
Challenges Along the Way
Like every real project, this competition involved several debugging sessions.
I encountered issues related to:
- preprocessing
- pipeline construction
- model experimentation
- evaluation
- submission workflow
Although these challenges were sometimes frustrating, solving them significantly improved my understanding of the complete machine learning lifecycle.
In hindsight, those debugging sessions taught me as much as the successful model training itself.
Beyond a Kaggle Score
One of the biggest misconceptions about Kaggle is that success is measured only by leaderboard position.
For me, the real achievement wasn't just generating predictions.
It was learning how to build an end-to-end machine learning workflow.
This project required me to combine concepts that I had previously learned separately:
- Exploratory Data Analysis
- Data Cleaning
- Feature Engineering
- Missing Value Handling
- Encoding
- Scaling
- Pipelines
- Ensemble Learning
- Model Evaluation
For the first time, these concepts came together in a single project.
Final Thoughts
This competition marked an important milestone in my machine learning journey.
Before this project, preprocessing techniques, pipelines, ensemble models, and evaluation metrics were topics I had studied individually.
Working through a real Kaggle competition showed me how these pieces fit together to solve an end-to-end machine learning problem.
While there is still plenty to learn—from advanced feature engineering and hyperparameter optimization to model ensembling and deployment—this experience gave me confidence that I can move beyond tutorials and apply machine learning concepts in practical settings.
If you're just beginning your Kaggle journey, my biggest advice is simple:
Don't chase the leaderboard first. Chase understanding.
A solid foundation in data analysis, preprocessing, and experimentation will take you much further than memorizing model parameters. Every competition is an opportunity to learn, refine your workflow, and become a better machine learning practitioner.
For me, this wasn't just another notebook—it was the project that transformed weeks of studying into real, hands-on experience. And that's exactly what makes Kaggle such a valuable platform for anyone serious about learning machine learning.
Top comments (0)