Every machine learning project starts with data, but the way you divide that data decides whether your results mean anything. A model that scores 98 percent in training may fail badly on new examples. This guide explains what each split does, how to choose split ratios, where leakage creeps in, and how to use cross-validation when data is scarce.
What is a train, validation, and test split?
A train, validation, and test split divides one dataset into three separate subsets, and each subset has a different job. The training set fits the model parameters, the validation set tunes choices such as hyperparameters, and the test set gives one final unbiased score on examples the model has never seen. Keeping the three jobs separate is what makes the final score trustworthy.
The training set is the largest part of the data and the only part the model learns from directly. During training, the algorithm adjusts its internal parameters to reduce error on these examples. The validation set is never used to fit parameters. Instead, it is used to compare choices: which value of k works best in k-nearest neighbors, how deep a tree should grow, or when to stop training before the model starts memorizing noise. The test set stays untouched until the very end, when the final model is scored on it once for an honest estimate of performance on new data.
This three-way division is the foundation of honest evaluation in supervised learning. A clear explanation of training, validation, and test data shows how the three subsets work together: training fits, validation tunes, and testing measures. If you are still building your mental model of the field, a plain overview of what is machine learning is a useful starting point.
Why do you need three splits instead of two?
Two splits are not enough, because tuning on the test set quietly turns it into a second validation set and destroys the unbiased final score. Every time you adjust a model based on test results, the test set stops being unseen data and starts being part of the training process. Three splits exist to prevent exactly that.
You train a model and check it on the validation set. The score tells you the model is underfitting, so you increase its capacity and check again. Each tuning choice is guided by the validation set. By the end of this process, the model has been shaped by the validation set, even though it never trained on it directly. The validation score is therefore optimistic, because it reflects a model tuned to do well on that particular set of examples.
The test set is the only protection against that optimism, and it works only if it stays untouched until tuning is finished. The moment you use test results to change a hyperparameter or pick between two candidate models, the test set has become another validation set, and you no longer have an unbiased estimate of real-world performance. The rule is simple and strict: the test set is opened once, at the end, for a single final measurement.
The third split also matters for detecting overfitting. A model that memorizes its training data will show a large gap between training accuracy and validation accuracy, and that gap signals overfitting. Understanding overfitting in machine learning helps you read that gap correctly: a small gap with low scores on both sets points to underfitting, while a large gap with a high training score points to memorization rather than learning.
What goes wrong when the test set leaks?
Leakage means information from the test set influences training or tuning, so the test score looks better than the model really is. The model appears accurate in evaluation and then underperforms on genuinely new data, because the evaluation was never honest in the first place.
The most common form of leakage is preprocessing the full dataset before splitting it. If you compute a scaler, an imputer, or a text vectorizer on all the data and then split, the training set has already seen statistics from the test set. The correct order is to split first and fit every preprocessing step on the training set only, then apply the same fitted transformation to the validation and test sets. The same applies to feature selection on the whole dataset, which also inflates the final score.
Duplicate rows are another quiet source of leakage. Near-duplicates mean a random split can place one copy in the training set and another in the test set. The model then appears to generalize when it is really recognizing examples it has seen. Deduplication before splitting, or grouping related rows so that all copies land in the same subset, prevents this.
Time-ordered data leaks differently. If your data has a time dimension, such as sales records or sensor readings, a random split mixes past and future. The model can then learn from future examples to predict past ones, which cannot happen in production. Time-based data needs a chronological split: earlier data for training, later data for validation, and the latest data for testing.
How do you choose split ratios like 80/10/10?
Common ratios are 80/10/10 or 70/15/15, and the right choice depends on dataset size, the number of classes, and how much tuning you plan to do. There is no single ratio that fits every project, but the trade-offs are straightforward.
The training set needs to be large enough for the model to learn the pattern. With a small dataset, a 60/20/20 split may leave too little training data, and the model underperforms for lack of examples. With a large dataset, an 80/10/10 split still leaves plenty of examples for validation and testing while giving the model most of the data to learn from. As a rule of thumb, the training share grows with dataset size, while the validation and test shares can shrink in percentage terms.
The validation set needs to be large enough to distinguish between candidate models. If you are comparing ten hyperparameter settings and the validation set has only fifty examples, the differences between settings will be noise. The test set needs the same: enough examples for a meaningful final score. A test set of twenty examples cannot support a serious claim about accuracy, because a single example changes the score by five percentage points.
Class balance changes the calculation. When one class is rare, a random split can leave the validation or test set with almost no examples of that class. Stratified splitting fixes this by preserving the class proportions in every subset, so each split contains a representative mix. Stratification should be the default whenever the target is a category rather than a number.
What is cross-validation and when should you use it?
Cross-validation rotates the validation role across folds so that every example is used for both training and validation, which gives a more stable estimate when data is scarce. Instead of relying on a single validation split, you divide the data into k folds, train on k minus one folds, validate on the remaining fold, and repeat until each fold has served as the validation set once. The average of the k scores is more reliable than any single split.
The most common form is k-fold cross-validation with k set to five or ten. Five folds is a good balance between stability and compute cost, since it requires training five models instead of one. Ten folds is slightly more stable at twice the cost. Stratified k-fold preserves class proportions in each fold and is the right default for classification problems. For time-ordered data, each validation fold must come after its training folds in time.
Cross-validation is most valuable when the dataset is small. With only a few hundred examples, a single validation split may be unrepresentative, and the ranking of candidate models can flip depending on which examples happen to land in the validation set. It is also the standard way to tune hyperparameters, since the average across folds is a fairer basis for comparing settings.
Cross-validation does not replace the test set. The folds are used for tuning, which means the cross-validated score is still an optimistic estimate, just a less noisy one. The honest final measurement still comes from a held-out test set that was never used in any fold. The scikit-learn documentation on cross-validation describes the standard patterns, including k-fold, stratified, and time series splits, and it is a solid reference for the exact behavior of each variant.
How do you split data for different kinds of models?
The split strategy changes with the task, because different kinds of models face different risks. A single random split is fine for some problems and misleading for others, so the choice should follow the structure of the data.
For ordinary supervised learning on tabular data, a random or stratified split is usually correct, provided the examples are independent of each other. Well-known benchmarks such as MNIST, CIFAR-10, and ImageNet ship with fixed splits so that results from different teams can be compared fairly. When you use a benchmark, keep its official split instead of inventing your own, because changing it makes your numbers incomparable with published results.
For time series and other ordered data, the split must be chronological. Random shuffling would let the model peek at the future, so the training period comes first, the validation period second, and the test period last. For grouped data, such as multiple records from the same patient, customer, or device, the split must keep each group whole. Splitting by group ensures the model is tested on groups it has never seen, which honestly measures generalization to new groups.
For clustering and unsupervised learning there is no labeled target to stratify on, but splits are still useful: fitting a clustering on one subset and checking its stability on another tells you whether the clusters are real structure or an artifact of one particular sample. For large language model fine-tuning, a held-out set of prompts and expected outputs, kept separate from the fine-tuning data, is the only way to tell whether the tuned model improved or simply memorized its training prompts.
What are the most common splitting mistakes?
The most common splitting mistakes are preprocessing before splitting, tuning on the test set, using random splits for time-ordered data, ignoring class balance, and reusing the test set after peeking at its results. Each of these is easy to make and easy to prevent once you know to look for it.
Preprocessing before splitting is the mistake that appears most often in tutorials and starter code, because fitting a scaler on the full dataset is one line of code and looks harmless. The fix is mechanical: split first, fit preprocessing on the training set only, and apply the fitted transformer to the other sets. Any step that learns something from the data must learn it from the training set alone.
Tuning on the test set is the mistake that costs the most credibility, because it invalidates the only unbiased measurement you have. The fix is procedural: decide in advance that the test set is opened once, at the end. If the test score disappoints, the honest response is to improve the model and re-tune on the validation set, not to adjust the model until the test score looks good.
Chronological splits fix ordered data, and stratified splits fix class balance. But reusing a test set across many experiments slowly turns it into a validation set, because each experiment leaks a little information about it. When a test set has guided many decisions, the right move is to collect a fresh held-out set or accept that the reported score is optimistic.
Conclusion
Train, validation, and test splits are a simple idea with strict rules, and those rules are what make model evaluation honest. The training set fits the model, the validation set guides tuning, and the test set gives one final unbiased score. Leakage breaks the separation at any of these steps. Sensible ratios, stratification, chronological splits, and cross-validation keep the evaluation trustworthy. A model evaluated this way may score lower than a leaky one, but its score means something, and that is the whole point of the split.
FAQs
Can I use the test set more than once?
No, not if you want the score to stay unbiased. Opening the test set once for a final measurement is the rule. Every additional use turns it further into a validation set. If you need to evaluate again after changing the model, re-tune on the validation set instead.
How big should the validation set be?
It should be large enough to rank candidate models reliably. For most projects, 10 to 20 percent of the data is a reasonable validation share. If the validation scores of competing models differ by less than the noise in the measurement, the validation set is too small to decide between them.
Is cross-validation a replacement for a test set?
No. Cross-validation improves the tuning process by averaging across folds, but the folds are still used to make decisions, so the cross-validated score is optimistic. A held-out test set that never took part in any fold is still needed for the final unbiased measurement.
What should I do if my dataset is very small?
Use cross-validation for tuning so that every example contributes to both training and validation, and keep a small held-out test set for the final check even if it is only a few dozen examples. With very small data, the test score will be noisy, so report it alongside the cross-validated scores.
Top comments (0)