Kaggle Titanic Practice 7: Overcoming Class 3 Deadlocks with Ticket Neighbor Survival OOF

#kaggle #machinelearning #python #catboost

Abstract

Identified a model blind spot through error analysis showing that misclassified samples for Class 3 passengers made up approximately 57% of the total errors.
Achieved CV accuracy of 0.8687, but Public Score dropped to 0.78947.

Finding the Model's Blind Spot

Using our optimized CatBoost model from the previous runs, we performed a detailed error analysis on the misclassified samples.
The analysis revealed that 56.9% of the 130 misclassified samples were Class 3 (Pclass 3) passengers, specifically male passengers.

Error Analysis (Pclass distribution of misclassified samples)

Reconstructing Cabin Layouts via Ticket Proximity

While the Cabin number feature is missing for approximately 77% of the passengers, the physical room location on the ship was a critical factor for survival. In other words, which cabin area passengers stayed in directly relates to the distance to the lifeboat deck and the escape routes from the flooded areas.
Thus, we focused on the numerical sequence of the Ticket numbers. Passengers with adjacent ticket numbers (difference in the last digits <= 5) likely purchased their tickets together and were assigned adjacent cabins in the same deck sector.
By feeding the "survival rate of adjacent ticket neighbors (who stayed in the same physical area)" to the model, it can capture whether that specific boarding sector was favorable for evacuation or was a deadlock zone.

Ticket Proximity OOF Distribution

Based on this domain knowledge, we engineered two key features:

Ticket Neighbor Survival (OOF_Ticket_Neighbor_Survival): We calculated the average survival rate of adjacent ticket neighbors (excluding the passenger themselves). To prevent target leakage, we computed this using the Out-of-Fold (OOF) method.
Class 3 Prefix Interaction (Prefix_[prefix]_3rd): We created interaction features between major ticket prefixes (a5, pc, ca, stono, sotono2) and Pclass 3 to capture specific boarding sectors.

# Calculation logic for Ticket Neighbor Survival (OOF)
cv_for_oof = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_neighbor_survival = pd.Series(0.5, index=train_fe.index)

for train_idx, val_idx in cv_for_oof.split(train_fe, y_train):
    tr_df = train_fe.iloc[train_idx].copy()
    tr_df['Survived'] = y_train.iloc[train_idx]

    for idx in val_idx:
        row = train_fe.iloc[idx]
        t_num = row['Ticket_Num']
        if not pd.isna(t_num):
            # Extract adjacent ticket neighbor samples within +-5
            neighbors = tr_df[
                (tr_df['Ticket_Num'] >= t_num - 5) & 
                (tr_df['Ticket_Num'] <= t_num + 5) & 
                (tr_df['PassengerId'] != row['PassengerId'])
            ]
            if len(neighbors) > 0:
                oof_neighbor_survival.loc[idx] = neighbors['Survived'].mean()
            else:
                p_s_mean = tr_df[(tr_df['Pclass'] == row['Pclass']) & (tr_df['Sex_male'] == row['Sex_male'])]['Survived'].mean()
                oof_neighbor_survival.loc[idx] = p_s_mean
        else:
            p_s_mean = tr_df[(tr_df['Pclass'] == row['Pclass']) & (tr_df['Sex_male'] == row['Sex_male'])]['Survived'].mean()
            oof_neighbor_survival.loc[idx] = p_s_mean

Validation Results and Kaggle Submission

We ran our evaluation script to compare multiple patterns using 5-Fold Stratified CV.

evaluate_step8.py (Full code of the temporary validation script)

import pandas as pd
import numpy as np
from sklearn.model_selection import StratifiedKFold
from sklearn.ensemble import RandomForestRegressor
from catboost import CatBoostClassifier
import optuna
import os

# --- 1. Data Loading and Basic Processing ---
print("--- Loading Data and Basic Processing ---")
train = pd.read_csv('data/raw/train.csv')
test = pd.read_csv('data/raw/test.csv')

df_all = pd.concat([train, test], sort=False).reset_index(drop=True)
df_all['Last_Name'] = df_all['Name'].apply(lambda x: x.split(',')[0])

# Family Survival setup
DEFAULT_SURVIVAL_VALUE = 0.5
df_all['Family_Survival'] = DEFAULT_SURVIVAL_VALUE

for grp, grp_df in df_all.groupby(['Last_Name', 'Fare']):
    if len(grp_df) &gt; 1:
        for ind, row in grp_df.iterrows():
            smax = grp_df.drop(ind)['Survived'].max()
            smin = grp_df.drop(ind)['Survived'].min()
            passID = row['PassengerId']
            if smax == 1.0:
                df_all.loc[df_all['PassengerId'] == passID, 'Family_Survival'] = 1.0
            elif smin == 0.0:
                df_all.loc[df_all['PassengerId'] == passID, 'Family_Survival'] = 0.0

for grp, grp_df in df_all.groupby('Ticket'):
    if len(grp_df) &gt; 1:
        for ind, row in grp_df.iterrows():
            passID = row['PassengerId']
            if df_all.loc[df_all['PassengerId'] == passID, 'Family_Survival'].values[0] == 0.5:
                smax = grp_df.drop(ind)['Survived'].max()
                smin = grp_df.drop(ind)['Survived'].min()
                if smax == 1.0:
                    df_all.loc[df_all['PassengerId'] == passID, 'Family_Survival'] = 1.0
                elif smin == 0.0:
                    df_all.loc[df_all['PassengerId'] == passID, 'Family_Survival'] = 0.0

df_all['Title'] = df_all['Name'].str.extract(r' ([A-Za-z]+)\.', expand=False)
title_map = {'Mr':'Mr','Miss':'Miss','Mrs':'Mrs','Master':'Master','Dr':'Rare','Rev':'Rare','Col':'Rare','Major':'Rare','Mlle':'Miss','Countess':'Rare','Ms':'Miss','Lady':'Rare','Jonkheer':'Rare','Don':'Rare','Dona':'Rare','Mme':'Mrs','Capt':'Rare','Sir':'Rare'}
df_all['Title'] = df_all['Title'].map(title_map).fillna('Rare')

df_all['Fare'] = df_all['Fare'].fillna(df_all['Fare'].median())
df_all['Embarked'] = df_all['Embarked'].fillna(df_all['Embarked'].mode()[0])
df_all['Deck'] = df_all['Cabin'].fillna('U').apply(lambda x: x[0])
df_all['FamilySize'] = df_all['SibSp'] + df_all['Parch'] + 1
df_all['IsAlone'] = (df_all['FamilySize'] == 1).astype(int)

# Age Imputation
age_features = ['Pclass', 'Sex', 'SibSp', 'Parch', 'Fare', 'Embarked', 'Title', 'Deck', 'FamilySize', 'IsAlone', 'Age']
df_age_prep = df_all[age_features].copy()
cat_cols_for_age = ['Sex', 'Embarked', 'Title', 'Deck']
df_age_encoded = pd.get_dummies(df_age_prep, columns=cat_cols_for_age, drop_first=True)
train_age = df_age_encoded[df_age_encoded['Age'].notnull()]
test_age = df_age_encoded[df_age_encoded['Age'].isnull()]
X_train_age = train_age.drop(columns=['Age'])
y_train_age = train_age['Age']
X_test_age = test_age.drop(columns=['Age'])

age_regressor = RandomForestRegressor(n_estimators=100, random_state=42)
age_regressor.fit(X_train_age, y_train_age)
predicted_ages = age_regressor.predict(X_test_age)
df_all.loc[df_all['Age'].isnull(), 'Age'] = predicted_ages

# Advanced Group features
df_all['Ticket_Group_Size'] = df_all.groupby('Ticket')['PassengerId'].transform('count')
df_all['Group_Id'] = df_all['Ticket']
mask = df_all['Ticket_Group_Size'] == 1
df_all.loc[mask, 'Group_Id'] = df_all.loc[mask, 'Last_Name'] + '_' + df_all.loc[mask, 'Fare'].astype(str)

df_all['Group_Size'] = df_all.groupby('Group_Id')['PassengerId'].transform('count')
df_all['Is_Female_or_Child'] = ((df_all['Sex'] == 'female') | (df_all['Age'] &lt; 16)).astype(int)
df_all['Group_Female_Child_Ratio'] = df_all.groupby('Group_Id')['Is_Female_or_Child'].transform('mean')
df_all['Group_Mean_Age'] = df_all.groupby('Group_Id')['Age'].transform('mean')
pclass_fare_median = df_all.groupby('Pclass')['Fare'].transform('median')
df_all['Group_Fare_Median_Diff'] = df_all['Fare'] - pclass_fare_median

df_all = df_all.drop(columns=['Is_Female_or_Child'])


# --- 2. New Feature Engineering ---
df_all['Fare_per_person'] = df_all['Fare'] / df_all['Ticket_Group_Size']

# Step 1: Socio-Physical Class Features
fare_threshold_3rd = df_all[df_all['Pclass'] == 3]['Fare_per_person'].quantile(0.1)
df_all['Is_Ultra_Poor_3rd'] = ((df_all['Pclass'] == 3) &amp; (df_all['Fare_per_person'] &lt;= fare_threshold_3rd)).astype(int)
df_all['Embarked_S_3rd'] = ((df_all['Pclass'] == 3) &amp; (df_all['Embarked'] == 'S')).astype(int)
df_all['Embarked_C_3rd'] = ((df_all['Pclass'] == 3) &amp; (df_all['Embarked'] == 'C')).astype(int)
df_all['Embarked_Q_3rd'] = ((df_all['Pclass'] == 3) &amp; (df_all['Embarked'] == 'Q')).astype(int)

# Step 2: Proximity Features
def extract_ticket_num(ticket):
    parts = ticket.split()
    if len(parts) == 0:
        return np.nan
    last_part = parts[-1]
    if last_part.isdigit():
        return int(last_part)
    return np.nan
df_all['Ticket_Num'] = df_all['Ticket'].apply(extract_ticket_num)

def extract_ticket_prefix(ticket):
    parts = ticket.split()
    if len(parts) &gt; 1:
        prefix = "".join(parts[:-1])
        prefix = prefix.replace(".", "").replace("/", "").lower()
        return prefix
    return 'none'
df_all['Ticket_Prefix'] = df_all['Ticket'].apply(extract_ticket_prefix)

major_prefixes = ['a5', 'pc', 'ca', 'stono', 'sotono2']
for pref in major_prefixes:
    col_name = f'Prefix_{pref}_3rd'
    df_all[col_name] = ((df_all['Pclass'] == 3) &amp; (df_all['Ticket_Prefix'] == pref)).astype(int)

# Step 3: Family Action Signals
df_all['Male_3rd_Has_Family'] = ((df_all['Pclass'] == 3) &amp; (df_all['Sex'] == 'male') &amp; (df_all['Age'] &gt;= 16) &amp; (df_all['SibSp'] + df_all['Parch'] &gt; 0)).astype(int)
df_all['Group_Has_Child'] = df_all.groupby('Group_Id')['Age'].transform(lambda x: (x &lt; 16).any()).astype(int)
df_all['Is_3rd_Parent_Guardian'] = ((df_all['Pclass'] == 3) &amp; (df_all['Age'] &gt;= 16) &amp; (df_all['Group_Has_Child'] == 1)).astype(int)

df_all = df_all.drop(columns=['Group_Has_Child', 'Ticket_Group_Size'])


# --- 3. Feature Set Definition ---
base_features = [
    'Pclass', 'Age', 'SibSp', 'Parch', 'Fare', 'Family_Survival', 'FamilySize', 'IsAlone',
    'Group_Size', 'Group_Female_Child_Ratio', 'Group_Mean_Age', 'Group_Fare_Median_Diff',
    'Sex_male', 'Embarked_Q', 'Embarked_S', 'Title_Miss', 'Title_Mr', 'Title_Mrs', 'Title_Rare',
    'Deck_B', 'Deck_C', 'Deck_D', 'Deck_E', 'Deck_F', 'Deck_G', 'Deck_T', 'Deck_U'
]
features_pattern_a = ['Is_Ultra_Poor_3rd', 'Embarked_S_3rd', 'Embarked_C_3rd', 'Embarked_Q_3rd']
features_pattern_c = ['Male_3rd_Has_Family', 'Is_3rd_Parent_Guardian']


# --- 4. Dummy Variable Encoding ---
cat_cols = ['Sex', 'Embarked', 'Title', 'Deck']
df_encoded = pd.get_dummies(df_all, columns=cat_cols, drop_first=True)

train_fe = df_encoded.iloc[:len(train)].copy()
test_fe = df_encoded.iloc[len(train):].copy()
y_train = train_fe['Survived'].astype(int)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

def get_objective(X_tr_full, y_tr):
    def objective(trial):
        params = {
            'iterations': trial.suggest_int('iterations', 50, 300),
            'depth': trial.suggest_int('depth', 3, 8),
            'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.15),
            'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', 1, 10),
            'verbose': 0,
            'random_seed': 42
        }
        model = CatBoostClassifier(**params)
        scores = []
        for train_idx, val_idx in cv.split(X_tr_full, y_tr):
            X_tr, y_tr_fold = X_tr_full.iloc[train_idx], y_tr.iloc[train_idx]
            X_va, y_va_fold = X_tr_full.iloc[val_idx], y_tr.iloc[val_idx]
            model.fit(X_tr, y_tr_fold)
            preds = model.predict(X_va)
            scores.append(np.mean(preds == y_va_fold))
        return np.mean(scores)
    return objective

optuna.logging.set_verbosity(optuna.logging.WARNING)
results = {}

# [Baseline]
print("\n--- Evaluating Baseline ---")
X_train_base = train_fe[base_features]
study = optuna.create_study(direction='maximize')
study.optimize(get_objective(X_train_base, y_train), n_trials=20)
results['Baseline'] = {'accuracy': study.best_value, 'params': study.best_params, 'features': base_features}

# [Pattern A]
print("\n--- Evaluating Pattern A ---")
features_a = base_features + features_pattern_a
X_train_a = train_fe[features_a]
study = optuna.create_study(direction='maximize')
study.optimize(get_objective(X_train_a, y_train), n_trials=20)
results['Pattern A'] = {'accuracy': study.best_value, 'params': study.best_params, 'features': features_a}

# [Pattern B]
print("\n--- Evaluating Pattern B ---")
oof_neighbor_survival = pd.Series(0.5, index=train_fe.index)
for train_idx, val_idx in cv.split(train_fe, y_train):
    tr_df = train_fe.iloc[train_idx].copy()
    tr_df['Survived'] = y_train.iloc[train_idx]
    for idx in val_idx:
        row = train_fe.iloc[idx]
        t_num = row['Ticket_Num']
        if not pd.isna(t_num):
            neighbors = tr_df[(tr_df['Ticket_Num'] &gt;= t_num - 5) &amp; (tr_df['Ticket_Num'] &lt;= t_num + 5) &amp; (tr_df['PassengerId'] != row['PassengerId'])]
            if len(neighbors) &gt; 0:
                oof_neighbor_survival.loc[idx] = neighbors['Survived'].mean()
            else:
                oof_neighbor_survival.loc[idx] = tr_df[(tr_df['Pclass'] == row['Pclass']) &amp; (tr_df['Sex_male'] == row['Sex_male'])]['Survived'].mean()
        else:
            oof_neighbor_survival.loc[idx] = tr_df[(tr_df['Pclass'] == row['Pclass']) &amp; (tr_df['Sex_male'] == row['Sex_male'])]['Survived'].mean()

train_fe_b = train_fe.copy()
train_fe_b['OOF_Ticket_Neighbor_Survival'] = oof_neighbor_survival
features_b = base_features + [f'Prefix_{pref}_3rd' for pref in major_prefixes] + ['OOF_Ticket_Neighbor_Survival']
X_train_b = train_fe_b[features_b]

study = optuna.create_study(direction='maximize')
study.optimize(get_objective(X_train_b, y_train), n_trials=20)
results['Pattern B'] = {'accuracy': study.best_value, 'params': study.best_params, 'features': features_b}

# [Pattern C]
print("\n--- Evaluating Pattern C ---")
features_c = base_features + features_pattern_c
X_train_c = train_fe[features_c]
study = optuna.create_study(direction='maximize')
study.optimize(get_objective(X_train_c, y_train), n_trials=20)
results['Pattern C'] = {'accuracy': study.best_value, 'params': study.best_params, 'features': features_c}

# [Pattern D]
print("\n--- Evaluating Pattern D ---")
train_fe_d = train_fe_b.copy()
train_fe_d['Is_Ultra_Poor_3rd'] = train_fe['Is_Ultra_Poor_3rd']
train_fe_d['Embarked_S_3rd'] = train_fe['Embarked_S_3rd']
train_fe_d['Embarked_C_3rd'] = train_fe['Embarked_C_3rd']
train_fe_d['Embarked_Q_3rd'] = train_fe['Embarked_Q_3rd']
train_fe_d['Male_3rd_Has_Family'] = train_fe['Male_3rd_Has_Family']
train_fe_d['Is_3rd_Parent_Guardian'] = train_fe['Is_3rd_Parent_Guardian']
features_d = features_b + features_pattern_a + features_pattern_c
X_train_d = train_fe_d[features_d]

study = optuna.create_study(direction='maximize')
study.optimize(get_objective(X_train_d, y_train), n_trials=20)
results['Pattern D'] = {'accuracy': study.best_value, 'params': study.best_params, 'features': features_d}

print("\n--- Final CV Comparison ---")
for key, val in results.items():
    print(f"{key}: {val['accuracy']:.5f}")

Here is the summary of validation results:

Baseline (Before features): CV 0.8530
Pattern A (Socio-Physical features only): CV 0.8530
Pattern B (Ticket Proximity OOF & Prefix Interaction): CV 0.8687 (Significant Improvement!)
Pattern C (Family & Guardian features only): CV 0.8574
Pattern D (All features combined): CV 0.8642

Pattern B using Ticket Proximity achieved a record-high CV of 0.8687.
We generated predictions using this optimal configuration and submitted them to Kaggle.
However, the Public Score remained at 0.78947, not showing an immediate increase on the leaderboard.

Conclusion

We are still far from satisfied, but resolving the physical cabin layouts through ticket number sequences has proven to be highly effective for CV improvement.
We hope this walkthrough helps.

Japanese version:
Kaggle Practice 1 "Titanic Survival Prediction" 1. Creating Kaggle Titanic Execution Environment on Local PC
Kaggle Practice 1 "Titanic Survival Prediction" 2. Initial Submission
Kaggle Practice 1 "Titanic Survival Prediction" 3. Cabin Feature Engineering
Kaggle Practice 1 "Titanic Survival Prediction" 4. Feature Engineering (Age Imputation via Random Forest)
Kaggle Practice 1 "Titanic Survival Prediction" 5. Feature Engineering (Nonlinear Transformation and Binning of Numerical Features)
Kaggle Practice 1 "Titanic Survival Prediction" 6. Adding Group Statistics to Capture Evacuation Behavior and CatBoost × Optuna Optimization
Kaggle Practice 1 "Titanic Survival Prediction" 7. Pseudo-restoration of Cabin Layouts via Ticket Proximity and Model Deadlock Analysis