DEV Community

Isaac Land
Isaac Land

Posted on Originally published at metho.scholay.com

How principal component analysis compresses correlated variables

How principal component analysis compresses correlated variables

The blue scatter cloud of the two-dimensional normal distribution forms a slanted ellipse, with two black arrows originating from the center, pointing respectively along the first principal component in the direction of the major axis and perpendicular to it along the second principal component.

Scatter plot of a bivariate normal distribution sample, with two arrows pointing respectively to the direction of maximum variance (first principal component) and to the direction orthogonal to it (second principal component). Author: Nicoguaro; Source:Wikimedia Commons; Permission:CC BY 4.0。

Principal Component Analysis (PCA) transforms a set of correlated numerical variables into a set of linear combinations that are uncorrelated, and orders them by variance from largest to smallest, using the first few principal components to summarize most of the variation in the data. It was proposed by Pearson in 1901 and named by Hotelling in 1933. This entry introduces its principles, how to interpret loadings and scores, how to select the number of principal components, its distinction from factor analysis, common misuses, and includes a computable R example and a scree plot.

From correlated variables to orthogonal principal components

Suppose there are p numerical variables X₁, …, Xₚ. PCA first finds a linear combination PC1 = a₁₁X₁ + … + a₁ₚXₚ, maximizing its variance under the constraint that the length of the coefficient vector is 1; the second principal component maximizes its variance under the condition of being uncorrelated with the first principal component, and so on, yielding at most p principal components. The review by Jolliffe and Cadima (2016) summarizes this as "constructing new variables sequentially maximizing variance while being mutually uncorrelated," which reduces to an eigenvalue-eigenvector problem of the covariance matrix or correlation matrix: the eigenvectors provide the combination coefficients, and the corresponding eigenvalues are the variances of the principal components. Geometrically, this is equivalent to rotating the coordinate axes to align with the directions of maximum elongation of the data cloud (see figure); the title of Pearson's 1901 paper was exactly "The best fitting straight line and plane to a system of points in space."

Before starting, one must first decide whether to use a covariance matrix or a correlation matrix. PCA is sensitive to scale; variables with larger variances will dominate the first principal component. When variables have different units or vastly different variances, it is typically necessary to standardize each variable, which is equivalent to performing PCA on a correlation matrix. The R prcomp documentation uses the USArrests dataset as an example: the variances of the variables differ by several orders of magnitude, so directly using prcomp(USArrests) is "inappropriate"; scale. = TRUE should be set instead.

Computationally, prcomp performs singular value decomposition on the centered (and standardized) data matrix, rather than eigen decomposition of the covariance matrix, yielding better numerical precision; variances are computed using N−1 as the divisor. It should also be noted that the sign of the eigenvectors is arbitrary, and different software versions, or even different builds of the same software, may invert entire columns; when interpreting results, only the relative signs of the coefficients within the same principal component should be considered.

Loadings, score, and number of principal components

'Loadings' have two usages in literature and software. One refers to the eigenvector itself (the rotation in R, with each column having a sum of squares equal to 1); the other is the eigenvector multiplied by the standard deviation of the corresponding principal component, and in standardized PCA it equals the correlation between the original variable and the principal component; the square of this value is the proportion of variance of that variable explained by the principal component. The UCLA OARC lecture notes use the latter. When reading someone's report, first confirm whether 'loadings' refer to the first or the second definition. Scores are the coordinates of each observation on the principal component, equal to the standardized data multiplied by the eigenvector matrix; the scores of different principal components are uncorrelated, and their variances equal the corresponding eigenvalues. The scores can be used for plotting, clustering, or as input for subsequent models.

Take USArrests as an example: it records arrests for murder, assault, and rape per 100,000 residents, along with the proportion of the population living in urban areas, for 50 U.S. states in 1973. After standardization, the four eigenvalues are 2.480, 0.990, 0.357, and 0.173, totaling 4 (number of variables), explaining 62.0%, 24.7%, 8.9%, and 4.3% of the variance respectively, with the first two components cumulatively explaining 86.8%. The coefficients of the three crime variables on PC1 are similar (−0.536, −0.583, −0.543), while the coefficient for the urban population proportion is smaller (−0.278), which can be interpreted as 'overall crime level'; PC2 is dominated by the urban population proportion (absolute coefficient value 0.873), which can be interpreted as 'urbanization level'. Converted to correlation coefficients, the urban population proportion has a correlation of approximately 0.87 with PC2, and the first two principal components collectively explain about 95% of its variance. These names are interpretations by the analyst, not conclusions derived from the data itself.

Retaining several principal components has no unique answer, and common rules fall into four categories:

  • Cumulative variance ratio: Stop when the cumulative explained variance reaches a pre-specified proportion (such as 80%); the proportion itself is a convention, not a theorem.
  • Kaiser criterion: Retain the components of the correlation matrix with eigenvalues greater than 1 (Kaiser, 1960), meaning that a principal component explains more variance than any single standardized variable.
  • Scree plot: Cattell (1966) suggested plotting the eigenvalues in sequence and truncating at the elbow where the curve transitions from a steep decline to a gradual flattening; the determination of the elbow is subjective.
  • Parallel analysis: Horn (1965) computed eigenvalues using randomly generated data of the same size, retaining only components whose observed eigenvalues exceed the eigenvalues from random data to eliminate the artificial elevation caused by sampling error.

On USArrests, these rules are not consistent: Kaiser's criterion retains only PC1 because PC2's eigenvalue of 0.990 is just short of 1; the scree plot levels off at the third principal component, and truncating before the bend point gives two; the cumulative proportion of 86.8% also supports retaining two. The rules are merely references, and the final decision depends on what the principal components are used for.

Scree plot: the black line decreases with increasing component sequence and gradually levels off, with a gray horizontal line indicating the Kaiser criterion and the threshold of parallel analysis.

Scree plot of factor analysis of the Boston Housing dataset: the black line represents the proportion of variance explained by each component, the gray horizontal lines correspond to the Kaiser criterion and the Horn parallel analysis, and the vertical lines indicate the cumulative variance proportion. Author: Sigbert; Source:Wikimedia Commons; Permission:CC BY-SA 3.0。

R example for the USArrests dataset

The following code only uses the stats and graphics packages that come built into R; USArrests is included with R and can be directly recalcuated. The output standard deviations, rotation matrix, and variance proportions are consistent with those provided on the ISLR tidymodels experiment page, except that the signs of some columns may be overall opposite.

# USArrests ships with base R: 50 US states, 1973
data(USArrests)

# Standardise variables (correlation-matrix PCA); prcomp uses SVD
pca = prcomp(USArrests, center = TRUE, scale. = TRUE)

summary(pca)              # standard deviations and proportion of variance
eig = pca$sdev^2         # eigenvalues
round(eig, 3)             # 2.480 0.990 0.357 0.173
round(pca$rotation, 3)    # eigenvectors; column signs are arbitrary

# Variable-component correlations (eigenvector * component sd)
round(cor(USArrests, pca$x), 3)

# Scores of the first states on PC1 and PC2
head(round(pca$x[, 1:2], 2))

# Scree plot with the Kaiser reference line
plot(eig, type = "b", pch = 19, xaxt = "n",
     xlab = "Principal component", ylab = "Eigenvalue",
     main = "Scree plot: USArrests (standardised)")
axis(1, at = seq_along(eig))
abline(h = 1, lty = 2, col = "grey40")

# Optional: parallel analysis (requires the psych package)
# psych::fa.parallel(USArrests, fa = "pc")
Enter fullscreen mode Exit fullscreen mode

After running, the cumulative variance proportions given by summary are sequentially 0.620, 0.868, 0.957, 1.000. On the scree plot, the eigenvalues drop sharply from the first to the second component, continue to decrease noticeably from the second to the third component, and then level off thereafter; the dashed line y = 1 closely passes just above the second point, intuitively illustrating the discrepancy between the Kaiser criterion and the elbow point judgment. Python users can use statsmodels' get_rdataset("USArrests") to download the same data (requires an internet connection), then standardize it and pass it to sklearn.decomposition.PCA; StandardScaler uses N rather than N−1 as the divisor, resulting in eigenvalues differing by a factor of N/(N−1), while the variance proportions and loadings directions remain unaffected.

Differences from factor analysis and common misuses

PCA and factor analysis both involve linear combinations of variables, and in SPSS they even share the same menu, but the models differ. PCA is a transformation of observed variables without an error term, aiming to retain as much total variance as possible using a small number of components, equivalent to assuming no unique variance. Common factor analysis, on the other hand, is a generative model: observed variables equal loadings multiplied by latent factors plus independent errors for each variable, where factors only explain the common variance among variables. The scikit-learn documentation places both within the same framework: when the error covariance is σ²I, it yields probabilistic PCA, and when it is a diagonal matrix diag(ψ₁,…,ψₚ), it corresponds to factor analysis, the latter allowing each variable to have different levels of noise.

In practice, principal component analysis (PCA) is used to reduce variables, construct composite indices, or perform dimensionality reduction for subsequent models; factor analysis is employed when it is believed that the correlations among items are driven by some underlying construct (such as anxiety or intelligence) and when measurement error is intended to be separated out, often accompanied by rotations such as varimax to facilitate interpretation. The results of the two methods are often similar, but not always, and Chapter 7 of Jolliffe's textbook specifically discusses their differences.

The most common misusage is treating correlation as causation. Principal components are merely an alternative representation of a correlational structure: PC1 of USArrests combines the three crime rates into a single component, showing that they rise and fall together across states, neither implying that urbanization causes crime nor proving the existence of a 'crime propensity' entity. Naming principal components is an act of interpretation; including principal component scores in regression yields coefficients that reflect only associations involving combinations of the original variables. The direction of principal components is determined by the available data, not pre-specified (Jolliffe and Cadima, 2016), and could change with a different sample, so causal inferences still depend on the study design.

The second type of misuse is applying PCA indiscriminately to categorical variables. PCA is based on variance and linear correlation and is suitable for numerical variables. Directly inserting gender or region coded as 1, 2, 3 into PCA assumes an ordered and equidistant relationship among categories, making the calculated 'variance' meaningless. For data primarily consisting of categorical variables, multiple correspondence analysis (MCA) can be used instead. The example in FactoMineR uses MCA to analyze categorical questionnaire responses from 300 tea consumers; Jolliffe and Cadima (2016) also introduce variants of PCA tailored for different data types. Furthermore, in predictive modeling, PCA can only be fitted on the training set and then applied to the validation set; otherwise, data leakage will occur.

References

  1. Pearson, K. (1901). LIII. On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2(11), 559–572. doi:10.1080/14786440109462720

  2. Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6), 417–441. doi:10.1037/h0071325

  3. Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20(1), 141–151. doi:10.1177/001316446002000116

  4. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185. doi:10.1007/BF02289447

  5. Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1(2), 245–276. doi:10.1207/s15327906mbr0102_10

  6. Jolliffe, I. T. (2002). Principal Component Analysis (2nd ed.). Springer. doi:10.1007/b98835

  7. Jolliffe, I. T., & Cadima, J. (2016). Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A, 374(2065), 20150202. doi:10.1098/rsta.2015.0202

  8. R Core Team. prcomp {stats}: Principal Components Analysis. R Documentation

  9. R Core Team. USArrests {datasets}: Violent Crime Rates by US State. R Documentation

  10. Hvitfeldt, E. 12 Unsupervised Learning. ISLR tidymodels Labs. Online version

  11. UCLA OARC. Principal Components (PCA) and Exploratory Factor Analysis (EFA) with SPSS. Online lecture notes

  12. scikit-learn developers. 2.5. Decomposing signals in components (matrix factorization problems). User Guide

  13. FactoMineR. Multiple Correspondence Analysis. Example page

  14. Research methodology Rituals can be mimicked, but understanding cannot be disguised.

  15. Information provided by Scholay Academic One-Stop Platform Organize and publish; if there is any infringement, please provide feedback for removal.

  16. Information synchronization published in EnterScholar, The Scholar Community in the Intelligent Age


Originally published at Metho.

Top comments (0)