DEV Community

amal org
amal org

Posted on

Data Analyst Guide: Mastering Cross-Validation: Why 80/20 Split is Wrong

Data Analyst Guide: Mastering Cross-Validation: Why 80/20 Split is Wrong

Business Problem Statement

In many real-world data analysis scenarios, the traditional 80/20 split for training and testing data is often used. However, this approach can lead to overfitting and poor model performance on unseen data. In this tutorial, we will demonstrate the importance of cross-validation in evaluating model performance and provide a step-by-step guide on how to implement it.

Let's consider a real-world scenario where we are building a predictive model to forecast sales for an e-commerce company. The company has a large dataset of historical sales data, and we want to evaluate the performance of our model using cross-validation. The ROI impact of using cross-validation is significant, as it can help us avoid overfitting and improve the overall performance of our model, resulting in more accurate sales forecasts and increased revenue.

Step-by-Step Technical Solution

Step 1: Data Preparation

We will use the pandas library to load and prepare our dataset. Let's assume we have a CSV file containing the historical sales data.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold

# Load the dataset
df = pd.read_csv('sales_data.csv')

# Preprocess the data
df = df.dropna()  # Remove rows with missing values
df = df.drop_duplicates()  # Remove duplicate rows

# Define the features and target variable
X = df.drop('sales', axis=1)
y = df['sales']
Enter fullscreen mode Exit fullscreen mode

Step 2: Analysis Pipeline

We will use the sklearn library to implement our analysis pipeline.

# Split the data into training and testing sets using 80/20 split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train a random forest regressor model on the training data
model = RandomForestRegressor(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Make predictions on the testing data
y_pred = model.predict(X_test)

# Evaluate the model performance using mean squared error
mse = mean_squared_error(y_test, y_pred)
print(f'Mean Squared Error (80/20 Split): {mse:.2f}')
Enter fullscreen mode Exit fullscreen mode

Step 3: Model/Visualization Code

We will use the matplotlib library to visualize the performance of our model.

import matplotlib.pyplot as plt

# Plot the actual vs predicted values
plt.scatter(y_test, y_pred)
plt.xlabel('Actual Sales')
plt.ylabel('Predicted Sales')
plt.title('Actual vs Predicted Sales (80/20 Split)')
plt.show()
Enter fullscreen mode Exit fullscreen mode

Step 4: Performance Evaluation using Cross-Validation

We will use the KFold class from sklearn to implement k-fold cross-validation.

# Define the number of folds
n_folds = 5

# Initialize the k-fold cross-validation object
kf = KFold(n_splits=n_folds, shuffle=True, random_state=42)

# Initialize the list to store the mean squared errors for each fold
mse_list = []

# Iterate over the folds
for train_index, test_index in kf.split(X):
    # Split the data into training and testing sets for the current fold
    X_train_fold, X_test_fold = X.iloc[train_index], X.iloc[test_index]
    y_train_fold, y_test_fold = y.iloc[train_index], y.iloc[test_index]

    # Train a random forest regressor model on the training data for the current fold
    model_fold = RandomForestRegressor(n_estimators=100, random_state=42)
    model_fold.fit(X_train_fold, y_train_fold)

    # Make predictions on the testing data for the current fold
    y_pred_fold = model_fold.predict(X_test_fold)

    # Evaluate the model performance using mean squared error for the current fold
    mse_fold = mean_squared_error(y_test_fold, y_pred_fold)
    mse_list.append(mse_fold)

# Calculate the average mean squared error across all folds
avg_mse = sum(mse_list) / n_folds
print(f'Average Mean Squared Error (K-Fold Cross-Validation): {avg_mse:.2f}')
Enter fullscreen mode Exit fullscreen mode

Step 5: Production Deployment

To deploy our model in production, we can use the joblib library to save the trained model and load it in our production environment.

import joblib

# Save the trained model
joblib.dump(model, 'random_forest_model.joblib')

# Load the saved model in production
loaded_model = joblib.load('random_forest_model.joblib')
Enter fullscreen mode Exit fullscreen mode

SQL Queries

To store and retrieve our data, we can use the following SQL queries:

-- Create a table to store the sales data
CREATE TABLE sales_data (
    id INT PRIMARY KEY,
    sales DATE,
    sales_amount FLOAT
);

-- Insert data into the sales_data table
INSERT INTO sales_data (id, sales, sales_amount)
VALUES (1, '2022-01-01', 100.0),
       (2, '2022-01-02', 120.0),
       (3, '2022-01-03', 110.0);

-- Retrieve the sales data
SELECT * FROM sales_data;
Enter fullscreen mode Exit fullscreen mode

Metrics/ROI Calculations

To calculate the ROI of our model, we can use the following metrics:

  • Mean Squared Error (MSE)
  • Mean Absolute Error (MAE)
  • Root Mean Squared Percentage Error (RMSPE) 
# Calculate the mean absolute error
mae = np.mean(np.abs(y_test - y_pred))

# Calculate the root mean squared percentage error
rmspe = np.sqrt(np.mean(np.square((y_test - y_pred) / y_test)))

print(f'Mean Absolute Error: {mae:.2f}')
print(f'Root Mean Squared Percentage Error: {rmspe:.2f}')
Enter fullscreen mode Exit fullscreen mode

Edge Cases

To handle edge cases, we can use the following techniques:

  • Data normalization: to handle outliers and skewed data
  • Feature engineering: to create new features that can handle edge cases
  • Model selection: to select a model that can handle edge cases 
# Normalize the data using standard scaling
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
Enter fullscreen mode Exit fullscreen mode

Scaling Tips

To scale our model, we can use the following techniques:

  • Distributed computing: to train our model on large datasets
  • Parallel processing: to speed up our model training and prediction
  • Model pruning: to reduce the size of our model and improve its performance 
# Use distributed computing to train our model
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from joblib import Parallel, delayed

# Define the number of cores to use
n_cores = 4

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Define the model training function
def train_model(X_train, y_train):
    model = RandomForestRegressor(n_estimators=100, random_state=42)
    model.fit(X_train, y_train)
    return model

# Train the model in parallel
models = Parallel(n_jobs=n_cores)(delayed(train_model)(X_train, y_train) for _ in range(n_cores))

# Make predictions on the testing data
y_pred = [model.predict(X_test) for model in models]

# Evaluate the model performance
mse = [mean_squared_error(y_test, y_pred_i) for y_pred_i in y_pred]
print(f'Mean Squared Error: {np.mean(mse):.2f}')
Enter fullscreen mode Exit fullscreen mode

By following these steps and using the provided code, we can master cross-validation and improve the performance of our models. Remember to always evaluate your model using cross-validation and to handle edge cases and scale your model as needed.

Top comments (0)