DEV Community: kito2718

Kaggle Titanic: Improving Survival Prediction with Random Forest Age Imputation

kito2718 — Thu, 02 Jul 2026 14:00:04 +0000

Kaggle Practice 1: Setting Up a Local Environment for the Kaggle Titanic Competition
Kaggle Practice 2: First Submission
Kaggle Practice 3: Feature Engineering for Cabin
Kaggle Practice 4: Feature Engineering (Imputing Age with Random Forest)

https://www.kaggle.com/c/titanic

Available on GitHub

Abstract

Changed age imputation from median values by title to predictive imputation using RandomForestRegressor.
The best 5-Fold CV (Cross-Validation) score improved from 0.8507 to 0.8519 (Logistic Regression).
Kaggle Public Score increased from 0.78708 to 0.78947.
Validation code is committed to GitHub: titanic_eda_20260702_2031_age_imputation.ipynb.

Overview

In the Kaggle Titanic: Machine Learning from Disaster competition, passenger Age is a critical factor for predicting survival.
Previously, we filled missing values with the median age of each passenger title (Mr, Miss, Mrs, Master, Rare). This time, we tried a more advanced approach: predicting the missing ages using a machine learning model (RandomForestRegressor) based on other features (Pclass, Sex, SibSp, Parch, Fare, Embarked, Deck).

Implementation

Here is the preprocessing code for the imputation. We trained RandomForestRegressor on passengers with known ages and predicted the missing values.

from sklearn.ensemble import RandomForestRegressor

# Features used for predicting Age
age_features = ['Pclass', 'Sex', 'SibSp', 'Parch', 'Fare', 'Embarked', 'Title', 'Deck', 'FamilySize', 'IsAlone', 'Age']
df_age_prep = df_all[age_features].copy()

# One-Hot Encoding for categorical features
cat_cols_for_age = ['Sex', 'Embarked', 'Title', 'Deck']
df_age_encoded = pd.get_dummies(df_age_prep, columns=cat_cols_for_age, drop_first=True)

# Split into known and unknown age datasets
train_age = df_age_encoded[df_age_encoded['Age'].notnull()]
test_age = df_age_encoded[df_age_encoded['Age'].isnull()]

X_train_age = train_age.drop(columns=['Age'])
y_train_age = train_age['Age']
X_test_age = test_age.drop(columns=['Age'])

# Train regressor and predict missing age
age_regressor = RandomForestRegressor(n_estimators=100, random_state=42)
age_regressor.fit(X_train_age, y_train_age)
predicted_ages = age_regressor.predict(X_test_age)

# Impute missing values in the original dataframe
df_all.loc[df_all['Age'].isnull(), 'Age'] = predicted_ages

Validation Results

Comparison of 5-Fold CV (Cross-Validation) accuracy across different models:

Model	Before (Median by Title)	After (Random Forest Imputation)	Difference
Logistic Regression	0.8507 +/- 0.0104	0.8519 +/- 0.0115	+0.0012
Random Forest	0.8204 +/- 0.0193	0.8249 +/- 0.0348	+0.0045
XGBoost	0.8215 +/- 0.0241	0.8226 +/- 0.0244	+0.0011
LightGBM	0.8496 +/- 0.0211	0.8485 +/- 0.0147	-0.0011

Logistic Regression achieved our personal best 5-Fold CV accuracy of 0.8519.
We also observed accuracy improvements in tree-based models like Random Forest and XGBoost.

Kaggle Submission Score

We predicted the test dataset using the updated Logistic Regression model and submitted it to Kaggle.
Our Public Score successfully improved from 0.78708 to 0.78947!
It is encouraging to see that the local CV score improvement translated directly to the Kaggle Public Score.

Summary & Next Steps

By estimating passenger age from other relevant features instead of using simple median values, the model could learn a more realistic passenger representation.
For our next attempt, we will target hyperparameter tuning (using Optuna) and model ensembling to achieve further improvements.

Hope this helps!

Kaggle Titanic: Cabin Feature Engineering (Is It Really Effective?)

kito2718 — Thu, 02 Jul 2026 13:56:46 +0000

https://www.kaggle.com/c/titanic

Available on GitHub

Abstract

Extracted the deck (floor) information from the first letter of the Cabin feature and created a new feature.
Summary of the evaluation using 5-Fold CV (Cross-Validation) on models like LightGBM.

Introduction

Continuing with the Kaggle Titanic competition.
In the Titanic dataset, the Cabin (cabin number) column has more than 70% missing values, so I had previously excluded it.
However, in seeking survival possibilities, the physical distance to lifeboat stations and the rate of flooding when sinking might have differed depending on the cabin deck. Therefore, I thought it could be a meaningful feature.

Implementation

I implemented a preprocessing function to extract the first letter of Cabin as the Deck feature, filling missing values with 'U' (Unknown).
The code is as follows:

def feature_engineering(df):
    df = df.copy()
    # Existing preprocessing (e.g. extracting Title, filling Age)
    df['Fare'] = df['Fare'].fillna(df['Fare'].median())
    df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])

    # Extract the first letter of Cabin as Deck
    df['Deck'] = df['Cabin'].fillna('U').apply(lambda x: x[0])

    le = LabelEncoder()
    for col in ['Sex', 'Embarked', 'Title', 'Deck']:
        df[col] = le.fit_transform(df[col])
    return df

After preprocessing, I encoded the categorical variables using LabelEncoder and added the Deck feature to the model inputs.

Results

Evaluation results using 5-Fold CV:

Model	Before (Baseline)	After (Cabin Added)	Difference
Logistic Regression	0.8014 +/- 0.0133	0.7991 +/- 0.0199	-0.0023
Random Forest	0.8227 +/- 0.0077	0.8148 +/- 0.0149	-0.0079
XGBoost	0.8181 +/- 0.0220	0.8227 +/- 0.0159	+0.0046
LightGBM	0.8350 +/- 0.0178	0.8361 +/- 0.0278	+0.0011

The validation score of LightGBM improved slightly from 0.8350 to 0.8361.
Since XGBoost, another tree-based model, also improved, the feature seems to have some effect.

Submitting to Kaggle

When I submitted the predictions to the Titanic competition, the public score dropped slightly from 0.77272 to 0.77033. The simple addition of this feature might be introducing noise due to overfitting.

Summary

Given the score drop after submission, I will next try creating group features by clustering the cabin information.

Hope this helps!

Japanese version:
Kaggle Practice 1: Setting up Kaggle Titanic Environment on a Local PC
Kaggle Practice 2: First Submission
Kaggle Practice 3: Feature Engineering for Cabin
Kaggle Practice 4: Feature Engineering (Imputing Age with Random Forest)

Kaggle Titanic: My First Submission (EDA, Feature Engineering, and Model Evaluation)

kito2718 — Thu, 02 Jul 2026 13:49:36 +0000

https://www.kaggle.com/c/titanic

Available on GitHub

Abstract

Participated in the Kaggle Titanic competition and performed Exploratory Data Analysis (EDA).
Performed basic feature engineering such as extracting title prefix (Title) and family size (FamilySize).
Compared four modeling approaches (Logistic Regression, Random Forest, XGBoost, and LightGBM) using 5-fold cross-validation..
LightGBM yielded the best CV score, resulting in a public leaderboard score of 0.77272.

Introduction

This is a continuation of my Kaggle Titanic journey.
In this post, we'll start by visualizing and analyzing variables likely related to survival (such as gender and passenger class) and handling missing values as part of the exploratory data analysis (EDA) process.
After that, we'll extract titles from passenger names, build new features like family size, and compare model performance using 5-Fold Cross-Validation (CV).

Exploratory Data Analysis (EDA)

The Titanic dataset contains 891 training rows and 418 test rows. Missing values are one of the first challenges in the Titanic dataset:

Cabin: 77.1% (mostly missing)
Age: 19.9% (about 20% missing)
Embarked: 0.2% (only 2 missing values)

Visualization

Let's plot basic variables against survival rate.

1. Survival Rate by Gender and Passenger Class

Looking at survival rates by gender and passenger class (Pclass) reveals strong insights. Female passengers had a significantly higher survival rate, and passengers in 1st class (Pclass = 1) had the highest chance of survival.

2. Age Distribution

Here is the histogram of age distribution, split by survival status. Younger passengers, especially infants and small children, tended to have higher survival rates. and we can observe specific age groups with higher mortality.

3. Correlation Map

Let's examine the correlation heatmap between numerical features. As expected, Pclass and Fare exhibit a strong negative correlation because higher-class tickets were generally more expensive.

Here is the formula for calculating correlation ($r$):
$$r = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n} (x_i - \bar{x})^2} \sqrt{\sum_{i=1}^{n} (y_i - \bar{y})^2}}$$

Simplifying this using covariance and standard deviation:
$$r = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}$$
Where $\text{Cov}(X, Y)$ is the covariance between $X$ and $Y$ (representing how they move together), and $\sigma_X, \sigma_Y$ are their respective standard deviations (representing the spread of the data).

Preprocessing and Feature Engineering

Based on our EDA findings, I wrote a feature_engineering function to prepare the data:

Extract Title: Extract honorifics (Mr, Miss, Mrs, Master, etc.) from Name using regular expressions, mapping rare titles to Rare.
Impute Age: Instead of using the global median, impute missing age values using the median age of each specific title group. This approach produces more realistic age estimates than using a single global median. (e.g., child-level ages for 'Master' and adult-level ages for 'Mr/Mrs').
FamilySize and IsAlone: Sum sibling/spouse count (SibSp) and parent/child count (Parch) plus one (for the passenger themselves) to create FamilySize. If FamilySize is 1, set the IsAlone flag to 1.
Additional imputations: Fill missing Fare with the median and Embarked with the mode.
Categorical Encoding: Encode Sex, Embarked, and Title using LabelEncoder.

Here is the code:

def feature_engineering(df):
    df = df.copy()
    # Extract Title from Name
    df['Title'] = df['Name'].str.extract(r' ([A-Za-z]+)\.', expand=False)
    title_map = {
        'Mr': 'Mr', 'Miss': 'Miss', 'Mrs': 'Mrs', 'Master': 'Master',
        'Dr': 'Rare', 'Rev': 'Rare', 'Col': 'Rare', 'Major': 'Rare',
        'Mlle': 'Miss', 'Countess': 'Rare', 'Ms': 'Miss', 'Lady': 'Rare',
        'Jonkheer': 'Rare', 'Don': 'Rare', 'Dona': 'Rare', 'Mme': 'Mrs',
        'Capt': 'Rare', 'Sir': 'Rare'
    }
    df['Title'] = df['Title'].map(title_map).fillna('Rare')

    # Impute missing Age with median of each Title group
    df['Age'] = df.groupby('Title')['Age'].transform(lambda x: x.fillna(x.median()))

    # Family-related features
    df['FamilySize'] = df['SibSp'] + df['Parch'] + 1
    df['IsAlone'] = (df['FamilySize'] == 1).astype(int)

    # Impute Fare and Embarked
    df['Fare'] = df['Fare'].fillna(df['Fare'].median())
    df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])

    # Label Encoding for categorical variables
    le = LabelEncoder()
    for col in ['Sex', 'Embarked', 'Title']:
        df[col] = le.fit_transform(df[col])
    return df

After preprocessing, we selected 10 features:
['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked', 'Title', 'FamilySize', 'IsAlone'].

Model Evaluation and Results

We compared four popular machine learning models using Stratified 5-Fold Cross-Validation (CV).

Here is the evaluation code:

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
models = {
    'Logistic Regression': LogisticRegression(max_iter=1000, random_state=42),
    'Random Forest':       RandomForestClassifier(n_estimators=100, random_state=42),
    'XGBoost':             xgb.XGBClassifier(n_estimators=100, random_state=42, eval_metric='logloss', verbosity=0),
    'LightGBM':           lgb.LGBMClassifier(n_estimators=100, random_state=42, verbosity=-1, max_depth=3, num_leaves=7, learning_rate=0.05)
}
results = {}
for name, model in models.items():
    scores = cross_val_score(model, X_train, y_train, cv=cv, scoring='accuracy')
    results[name] = scores
    print(f'{name:<25}: {scores.mean():.4f} +/- {scores.std():.4f}')

The average cross-validation scores are shown below:
| Model | 5-Fold CV Score (Accuracy) |
| :--- | :--- |
| Logistic Regression | 0.8014 +/- 0.0133 |
| Random Forest | 0.8227 +/- 0.0077 |
| XGBoost | 0.8181 +/- 0.0220 |
| LightGBM | 0.8350 +/- 0.0178 |

We plotted the CV score distributions using boxplots:

LightGBM achieved the best CV score of 0.8350.
Random Forest also performed well, achieving a score of 0.8227.

Feature Importance

Let's visualize the feature importances of our best LightGBM model:

The model relied heavily on Sex, Title, Fare, and Age, which aligns well with the patterns observed during EDA.

First Kaggle Submission

Using our best LightGBM model, we generated predictions for the test dataset and created submission.csv:

preds = best_model.predict(X_test)
submission = pd.DataFrame({'PassengerId': test['PassengerId'], 'Survived': preds})
submission.to_csv('../submissions/submission.csv', index=False)

Leaderboard Score: 0.77272.
This became my personal best score at the time.
It is common for public leaderboard scores to be slightly lower than local CV scores due to differences between the training folds and the hidden test set.

Summary

We walked through the EDA, feature engineering, and model evaluation workflow. Evaluating multiple models proved to be very effective.
In the next article, we'll tackle the heavily missing Cabin feature and see whether it can help us push the score even higher.

Setting Up a Local Environment for the Kaggle Titanic Competition

kito2718 — Thu, 02 Jul 2026 13:20:58 +0000

https://www.kaggle.com/c/titanic

Available on GitHub

Abstract

Set up a local development environment for the Kaggle Titanic competition.

Introduction

Kaggle Notebooks are convenient, but local development offers faster iteration, better IDE support, and easier version control with Git.
I've been participating in the Kaggle Titanic competition and trying to improve my score. However, logging into Kaggle every time I wanted to submit an experiment quickly became tedious.

Environment Summary

Surprisingly, all I really needed was Python.

Item	Value
Python	3.14.6
Virtual Env	`.venv` (venv)
Project Path	`51_googleantigravity\1st_`
pip	26.1.x

Pitfalls/Things to Keep in Mind

I ran into a small issue during setup, so I'll document it here.

Crucial Point:

Avoid using non-ASCII characters (such as Japanese characters) in the virtual environment path. C extension libraries (DLLs) such as scipy and scikit-learn failed to load properly when the path contained non-ASCII characters.

1. Step-by-Step Guide

1.1. Installing Python

Download and install from the Official Python Release Page.

Scroll down to the bottom of the page to find the installer.

Also, add Python to your PATH:

setx PATH "%PATH%;<Python_Installation_Folder>"

1.2. Creating the Project Folder

Create it anywhere you like on your PC.

mkdir 51_googleantigravity\1st_

1.3. Creating the Virtual Environment

Create a Python virtual environment:

# Move to the project folder
cd 51_googleantigravity\1st_
python.exe -m venv .venv

Structure after creation:

1st_/
└── .venv/
    ├── Scripts/
    │   ├── activate
    │   ├── python.exe
    │   └── pip.exe
    └── Lib/

1.4. Installing Packages

Install various packages required for the Python code:

python.exe -m pip install --upgrade pip pandas numpy scikit-learn matplotlib seaborn jupyter kaggle xgboost lightgbm

Major packages installed:

Package	Version	Purpose
pandas	3.0.3	Data manipulation
numpy	2.5.0	Numerical calculations
scikit-learn	1.9.0	Machine learning
matplotlib	3.11.0	Data visualization
seaborn	0.13.2	Statistical data visualization
xgboost	3.3.0	Gradient boosting
lightgbm	4.6.0	Gradient boosting
jupyter / notebook	7.6.0	Jupyter Notebook environment
kaggle	2.2.3	Kaggle CLI

1.5. Directory Structure

Create directories for datasets, notebooks, and outputs:

mkdir data\raw
mkdir data\processed
mkdir notebooks
mkdir submissions

Organized folder structure:

1st_/
├── .venv/
├── data/
│   ├── raw/          ← Kaggle raw data
│   └── processed/    ← Visualizations and preprocessed data
├── notebooks/        ← Notebook files
└── submissions/      ← Submission CSV files

1.6. Creating requirements.txt

python.exe -m pip freeze > requirements.txt

1.7. Creating the Notebook

Create notebooks/titanic_eda.ipynb with the following structure:

Cell	Content
1	Import libraries
2	Load data (train.csv / test.csv)
3	Exploratory Data Analysis (missing values, survival rate, age distribution, correlation heatmap)
4	Feature Engineering (extracting titles, imputing age, family size, etc.)
5	Model training
6	Generate submission file (output to submissions/submission.csv)

1.8. Launching Jupyter Notebook

Start the notebook server:

# Move to project directory
cd 51_googleantigravity\1st_
# Activate virtual environment
.venv\Scripts\activate
# Start notebook
jupyter notebook notebooks\titanic_eda.ipynb

Your browser should open automatically. If not, copy the URL displayed in the terminal and paste it into your browser.

1.9. Preparing Kaggle CLI

Go to Kaggle Settings -> "API Tokens" tab.
Click "Create Legacy API Key" under the Legacy API Credentials section.
Place the downloaded kaggle.json file at C:\Users\<your_username>\.kaggle\kaggle.json.
Run the following:

# Activate virtual environment
.venv\Scripts\activate
# Download Titanic data using Kaggle API
kaggle competitions download -c titanic -p data\raw
# Extract ZIP (using standard Windows tar command)
tar -xf data\raw\titanic.zip -C data\raw

Alternatively, you can download files manually:

Download the following 3 files from https://www.kaggle.com/c/titanic/data and place them in data\raw\:

train.csv

test.csv

gender_submission.csv

1.10. Submitting to Kaggle

kaggle competitions submit -c titanic -f submissions\submission.csv -m "Your comment"

That's it! You now have a fully functional local environment for experimenting with Kaggle Titanic and submitting results directly from your machine.