Don’t Let the Same Participant Leak Into Both Sides of Your Model

When building a model from repeated observations, it’s easy to get a cross-validation score that looks better than the model’s ability to handle new people.

The problem can be data leakage through the split.

In my postgraduate research, I compared Single-Task Learning (STL) and Multi-Task Learning (MTL) models for predicting schizophrenia symptoms using the CrossCheck dataset. It includes data from 75 participants, and that participant structure matters: if one person’s observations appear in both training and test sets, the model may benefit from patterns specific to that person.

That is not the same as predicting for someone it has never seen.

The split should match the intended use

A random row-level split treats every observation as independent. But repeated observations from the same participant are related. If the goal is to estimate performance on unseen participants, all observations from each participant should stay together in a fold.

GroupKFold does this by keeping each group—here, each participant—on one side of a split at a time.

from sklearn.model_selection import GroupKFold

cv = GroupKFold(n_splits=5)

for train_idx, test_idx in cv.split(X, y, groups=participant_ids):
    X_train, X_test = X[train_idx], X[test_idx]
    y_train, y_test = y[train_idx], y[test_idx]

    # Fit the model on X_train and y_train.
    # Evaluate it on X_test and y_test.

Here, X is the feature matrix, y is a target, and participant_ids contains the participant identifier for each row. The important part is not the number of folds; it is that the grouping matches the question being asked.

If the intended use is predicting future observations from people already in the dataset, a time-based split may be more appropriate. The evaluation design should reflect deployment—not just whichever split is easiest to implement.

Comparing separate and shared models

I compared two approaches.

With Single-Task Learning, a separate model is fitted for each symptom target. I used LassoCV, which applies L1 regularisation and selects its regularisation strength through cross-validation.

With Multi-Task Learning, targets are fitted together using MultiTaskLassoCV. This approach can encourage related targets to share features.

The distinction is about the modelling assumption:

• STL allows each target to select its own features.
• MTL encourages targets to use a shared feature structure.

Neither assumption is automatically better. Whether sharing helps depends on the targets, features and data.

Keep model selection inside the training data

There is another detail to watch: LassoCV and MultiTaskLassoCV perform their own cross-validation to select regularisation settings. If that internal validation splits observations from the same participant across folds, the outer GroupKFold alone does not fully address the grouping issue.

For a rigorous evaluation, model selection should also respect participant groups. One option is to use a grouped inner split for tuning and a separate grouped outer split for final evaluation. The outer test fold should not be used to choose model settings.

Conceptually, the process is:

1. Hold out a group of participants for the outer test fold.
2. Use only the remaining participants to select model settings.
3. Fit the chosen model on those training participants.
4. Evaluate once on the held-out participants.
5. Repeat with each outer fold.

This is more involved than calling a default cross-validation estimator, but it makes the evaluation closer to the intended question: how well does the approach generalise to participants it did not train on?

Don’t report a score without its context

A metric is only meaningful alongside the evaluation design. When describing a result, report:

• what each row represents;
• how participants were assigned to folds;
• how model settings were selected;
• which metric was used;
• whether the metric was calculated per target or across targets.

In this research context, predictions should not be presented as diagnoses. A model’s evaluation on one dataset does not establish clinical usefulness or performance on a different population.

The central lesson from this comparison is straightforward: before comparing STL with MTL, make sure the validation setup tests the kind of generalisation you care about. With repeated participant data, keeping each participant together can make the difference between measuring memorisation and measuring generalisation.