VARIANCE IN MACHINE LEARNING

Definition of Variance

Variance is a statistical measure that indicates how far the data values or a machine learning model's predictions are spread out from their mean (average). It measures the amount of variability or dispersion in a dataset.

In machine learning, variance refers to how much a model's predictions change when it is trained on different training datasets. A model with high variance is very sensitive to small changes in the training data and may overfit, while a model with low variance produces more stable predictions and generally generalizes better to unseen data.Types of Variance in Machine Learning

1. Data Variance
Definition

Data variance refers to the variation present in the dataset itself. It shows how much the feature values differ from one another.

Example

Consider students' marks:

Low Variance: 78, 79, 80, 81, 82
High Variance: 30, 55, 80, 95, 100
Applications
Data analysis
Data preprocessing
Feature engineering

2. Model Variance
Definition

Model variance measures how much a machine learning model's predictions change when trained on different training datasets.

Characteristics
Sensitive to changes in training data
May memorize training data
Usually leads to overfitting
Example

A deep decision tree often has high model variance because small changes in the training data can produce a very different tree.

3. High Variance
Definition

High variance occurs when a model learns the training data too well, including its noise, resulting in poor performance on unseen data.

Characteristics
Overfitting
High training accuracy
Low testing accuracy
Poor generalization
Example

A decision tree with many levels fits the training data perfectly but performs poorly on new data.

Problems
Poor prediction on unseen data
Sensitive to small data changes
Solutions
Increase training data
Apply regularization
Use cross-validation
Simplify the model
Use ensemble methods such as Random Forest

4. Low Variance
Definition

Low variance means the model's predictions remain consistent even when trained on different datasets.

Characteristics
Stable predictions
Good generalization
Less sensitive to data changes
Example

Linear Regression often has lower variance than a complex Decision Tree.

Advantages
Better performance on unseen data
More reliable predictions